Re: Problem - Bringing up the HBase cluster
Really surprising ! All of sudden the fresh restart worked like a charm. Master initialization error is gone and i see the HMaster webapp clean and good. Thank you very much for the help. I will continue my testing by adding more data nodes and i will post if i see any errors Thanks for the excellent support Jothikumar On Wed, Aug 22, 2012 at 11:24 AM, Stack wrote: > On Wed, Aug 22, 2012 at 11:16 AM, Jothikumar Ekanath > wrote: > > Do my configuration looks correct? do i need to set anything in the > > hbase-env.sh? I already set the manage_ZK to true, > > > > Yes. > > It must be something about your environment. > > Lets see your logs. > > St.Ack >
Re: Problem - Bringing up the HBase cluster
Hi Stack, For sure, hdfs is up? You can put and get files into it? I can see the namenode webapp, didn't try the file get and put part What is hbase-master and hbase-namenode? Are they supposed to be the same machine? DNS says they are? They are different nodes (VM's) configured hosts file to map the dns correctly Where is hbase-region1? Not same as hbase-master and hbase-namenode? It is a seperate VM , not same as the master and namenode This is writable? Yes, it is. Do my configuration looks correct? do i need to set anything in the hbase-env.sh? I already set the manage_ZK to true, Thanks Jothikumar On Wed, Aug 22, 2012 at 11:01 AM, Stack wrote: > On Wed, Aug 22, 2012 at 10:41 AM, Jothikumar Ekanath > wrote: > > Hi Stack, > > > > Ok, i will cleanup everything and start from fresh. This time i will add > one > > more data node > > > > so 1 hbase master and 2 regions. Zookeeper managed by hbase is started in > > region1. This is my configuration, i will start everything from the > scratch > > and will see. If i get the same error, i will send the logs for review > > > > > > > > > > hbase.rootdir > > hdfs://hadoop-namenode:9000/hbase > > The directory shared by RegionServers. > > > > For sure, hdfs is up? You can put and get files into it? > > > > > hbase.master > > hbase-master:6 > > The host and port that the HBase master runs > > at. > > > > What is hbase-master and hbase-namenode? Are they supposed to be the > same machine? DNS says they are? > > > > > > hbase.cluster.distributed > > true > > The mode the cluster will be in. Possible values are > > false: standalone and pseudo-distributed setups with managed Zookeeper > true: > > fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh) > > > > > > > > hbase.zookeeper.quorum > > hbase-region1 > > > > > Where is hbase-region1? Not same as hbase-master and hbase-namenode? > > > > > > > > > > > hbase.zookeeper.property.dataDir > > /apps/zookeeper > > Property from ZooKeeper's config zoo.cfg. The > directory > > where the snapshot is stored. > > > > > > > > This is writable? > > St.Ack > > > > > Thanks > > Jothikumar > > > > > > On Wed, Aug 22, 2012 at 8:59 AM, Stack wrote: > >> > >> On Wed, Aug 22, 2012 at 8:43 AM, Jothikumar Ekanath > > >> wrote: > >> > Hi, > >> > Thanks for the response, sorry i put this email in the dev > >> > space. > >> > My data replication is 2. and yes the region and master server > >> > connectivity > >> > is good > >> > > >> > Initially i started with 4 data nodes and 1 master, i faced the same > >> > problem. So i reduced the data nodes to 1 and wanted to test it. I see > >> > the > >> > same issue. I haven't tested the pseudo distribution mode, i can test > >> > that. > >> > But my objective is to test the full distributed mode and do some > >> > testing. > >> > I can send my configuration for review. Please let me know if i am > >> > missing > >> > any basic setup configuration. > >> > > >> > >> Be sure you start fresh. Did you run it standalone previous? Did you > >> reset hbase tmp dir? If not, try clearing /tmp before starting. If > >> still does not work, put up your logs where we can take a look see -- > >> and your config. > >> > >> St.Ack > > > > >
Re: Problem - Bringing up the HBase cluster
On Wed, Aug 22, 2012 at 10:41 AM, Jothikumar Ekanath wrote: > Hi Stack, > > Ok, i will cleanup everything and start from fresh. This time i will add one > more data node > > so 1 hbase master and 2 regions. Zookeeper managed by hbase is started in > region1. This is my configuration, i will start everything from the scratch > and will see. If i get the same error, i will send the logs for review > > > > > hbase.rootdir > hdfs://hadoop-namenode:9000/hbase > The directory shared by RegionServers. > For sure, hdfs is up? You can put and get files into it? > > hbase.master > hbase-master:6 > The host and port that the HBase master runs > at. > What is hbase-master and hbase-namenode? Are they supposed to be the same machine? DNS says they are? > > hbase.cluster.distributed > true > The mode the cluster will be in. Possible values are > false: standalone and pseudo-distributed setups with managed Zookeeper true: > fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh) > > > > hbase.zookeeper.quorum > hbase-region1 > Where is hbase-region1? Not same as hbase-master and hbase-namenode? > > > > hbase.zookeeper.property.dataDir > /apps/zookeeper > Property from ZooKeeper's config zoo.cfg. The directory > where the snapshot is stored. > > > This is writable? St.Ack > > Thanks > Jothikumar > > > On Wed, Aug 22, 2012 at 8:59 AM, Stack wrote: >> >> On Wed, Aug 22, 2012 at 8:43 AM, Jothikumar Ekanath >> wrote: >> > Hi, >> > Thanks for the response, sorry i put this email in the dev >> > space. >> > My data replication is 2. and yes the region and master server >> > connectivity >> > is good >> > >> > Initially i started with 4 data nodes and 1 master, i faced the same >> > problem. So i reduced the data nodes to 1 and wanted to test it. I see >> > the >> > same issue. I haven't tested the pseudo distribution mode, i can test >> > that. >> > But my objective is to test the full distributed mode and do some >> > testing. >> > I can send my configuration for review. Please let me know if i am >> > missing >> > any basic setup configuration. >> > >> >> Be sure you start fresh. Did you run it standalone previous? Did you >> reset hbase tmp dir? If not, try clearing /tmp before starting. If >> still does not work, put up your logs where we can take a look see -- >> and your config. >> >> St.Ack > >
Re: Problem - Bringing up the HBase cluster
Hi Stack, Ok, i will cleanup everything and start from fresh. This time i will add one more data node so 1 hbase master and 2 regions. Zookeeper managed by hbase is started in region1. This is my configuration, i will start everything from the scratch and will see. If i get the same error, i will send the logs for review hbase.rootdir hdfs://hadoop-namenode:9000/hbase The directory shared by RegionServers. hbase.master hbase-master:6 The host and port that the HBase master runs at. hbase.cluster.distributed true The mode the cluster will be in. Possible values are false: standalone and pseudo-distributed setups with managed Zookeeper true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh) hbase.zookeeper.quorum hbase-region1 hbase.zookeeper.property.dataDir /apps/zookeeper Property from ZooKeeper's config zoo.cfg. The directory where the snapshot is stored. Thanks Jothikumar On Wed, Aug 22, 2012 at 8:59 AM, Stack wrote: > On Wed, Aug 22, 2012 at 8:43 AM, Jothikumar Ekanath > wrote: > > Hi, > > Thanks for the response, sorry i put this email in the dev > space. > > My data replication is 2. and yes the region and master server > connectivity > > is good > > > > Initially i started with 4 data nodes and 1 master, i faced the same > > problem. So i reduced the data nodes to 1 and wanted to test it. I see > the > > same issue. I haven't tested the pseudo distribution mode, i can test > that. > > But my objective is to test the full distributed mode and do some > testing. > > I can send my configuration for review. Please let me know if i am > missing > > any basic setup configuration. > > > > Be sure you start fresh. Did you run it standalone previous? Did you > reset hbase tmp dir? If not, try clearing /tmp before starting. If > still does not work, put up your logs where we can take a look see -- > and your config. > > St.Ack >
Re: Problem - Bringing up the HBase cluster
If you have a single datanode with a replication of two, it will (basically) won't work, as it will try to replicate the blocks on two datanodes while there is only one available. Note that I'm speaking about datanodes (i.e. hdfs) and not region servers (i.e. hbase). pastebin the full logs with the region server, may be someone will have an idea of the root issue. But I think it's safer to start with the pseudo distributed, it's easier to setup and it's documented. A distributed config with a single node is not really standard, it's better to start with the easiest path imho. On Wed, Aug 22, 2012 at 5:43 PM, Jothikumar Ekanath wrote: > Hi, > Thanks for the response, sorry i put this email in the dev space. > My data replication is 2. and yes the region and master server connectivity > is good > > Initially i started with 4 data nodes and 1 master, i faced the same > problem. So i reduced the data nodes to 1 and wanted to test it. I see the > same issue. I haven't tested the pseudo distribution mode, i can test that. > But my objective is to test the full distributed mode and do some testing. I > can send my configuration for review. Please let me know if i am missing any > basic setup configuration. > > > On Wed, Aug 22, 2012 at 12:00 AM, N Keywal wrote: >> >> Hi, >> >> Please use the user mailing list (added at dest) for this type of >> questions instead of the dev list (now in bcc). >> >> It's a little bit strange to use the full distributed mode with a >> single region server. Is the Pseudo-distributed mode working? >> Check the number of datanodes vs. dfs.replication (default 3). If you >> have less datanodes then dfs.replication value, it won't work >> properly. >> Check as well that the region server is connected to the master. >> >> Cheers, >> >> >> >> On Wed, Aug 22, 2012 at 3:16 AM, kbmkumar wrote: >> > Hi, >> > I am trying to bring up a HBase cluster with 1 master and 1 one >> > region >> > server. I am using >> > Hadoop 1.0.3 >> > Hbase 0.94.1 >> > >> > Starting the hdfs was straight forward and i could see the namenode up >> > and >> > running successfully. But the problem is with Hbase. I followed all the >> > guidelines given in the Hbase cluster setup (fully distributed mode) and >> > ran >> > the start-hbase.sh >> > >> > It started the Master, Region server and zookeeper (in the region >> > server) as >> > per my configuration. But i am not sure the master is fully functional. >> > When >> > i try to connect hbase shell and create table, it errors out saying >> > PleaseHoldException- Master is initializing >> > >> > In UI HMaster status shows like this *Assigning META region (since >> > 18mins, >> > 39sec ago)* >> > >> > and i see the Hmaster logs are flowing with the following debug prints, >> > the >> > log file is full of below prints, >> > * >> > 2012-08-22 01:08:19,637 DEBUG >> > >> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: >> > Looked up root region location, >> > >> > connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@49586cbd; >> > serverName=hadoop-datanode1,60020,1345596463277 >> > 2012-08-22 01:08:19,638 DEBUG >> > >> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: >> > Looked up root region location, >> > >> > connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@49586cbd; >> > serverName=hadoop-datanode1,60020,1345596463277 >> > 2012-08-22 01:08:19,639 DEBUG >> > >> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: >> > Looked up root region location, >> > >> > connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@49586cbd; >> > serverName=hadoop-datanode1,60020,1345596463277* >> > >> > Please help me in debugging this. >> > >> > >> > >> > >> > >> > -- >> > View this message in context: >> > http://apache-hbase.679495.n3.nabble.com/Problem-Bringing-up-the-HBase-cluster-tp4019948.html >> > Sent from the HBase - Developer mailing list archive at Nabble.com. > >
Re: Problem - Bringing up the HBase cluster
On Wed, Aug 22, 2012 at 8:43 AM, Jothikumar Ekanath wrote: > Hi, > Thanks for the response, sorry i put this email in the dev space. > My data replication is 2. and yes the region and master server connectivity > is good > > Initially i started with 4 data nodes and 1 master, i faced the same > problem. So i reduced the data nodes to 1 and wanted to test it. I see the > same issue. I haven't tested the pseudo distribution mode, i can test that. > But my objective is to test the full distributed mode and do some testing. > I can send my configuration for review. Please let me know if i am missing > any basic setup configuration. > Be sure you start fresh. Did you run it standalone previous? Did you reset hbase tmp dir? If not, try clearing /tmp before starting. If still does not work, put up your logs where we can take a look see -- and your config. St.Ack
Re: Problem - Bringing up the HBase cluster
Hi, Thanks for the response, sorry i put this email in the dev space. My data replication is 2. and yes the region and master server connectivity is good Initially i started with 4 data nodes and 1 master, i faced the same problem. So i reduced the data nodes to 1 and wanted to test it. I see the same issue. I haven't tested the pseudo distribution mode, i can test that. But my objective is to test the full distributed mode and do some testing. I can send my configuration for review. Please let me know if i am missing any basic setup configuration. On Wed, Aug 22, 2012 at 12:00 AM, N Keywal wrote: > Hi, > > Please use the user mailing list (added at dest) for this type of > questions instead of the dev list (now in bcc). > > It's a little bit strange to use the full distributed mode with a > single region server. Is the Pseudo-distributed mode working? > Check the number of datanodes vs. dfs.replication (default 3). If you > have less datanodes then dfs.replication value, it won't work > properly. > Check as well that the region server is connected to the master. > > Cheers, > > > > On Wed, Aug 22, 2012 at 3:16 AM, kbmkumar wrote: > > Hi, > > I am trying to bring up a HBase cluster with 1 master and 1 one > region > > server. I am using > > Hadoop 1.0.3 > > Hbase 0.94.1 > > > > Starting the hdfs was straight forward and i could see the namenode up > and > > running successfully. But the problem is with Hbase. I followed all the > > guidelines given in the Hbase cluster setup (fully distributed mode) and > ran > > the start-hbase.sh > > > > It started the Master, Region server and zookeeper (in the region > server) as > > per my configuration. But i am not sure the master is fully functional. > When > > i try to connect hbase shell and create table, it errors out saying > > PleaseHoldException- Master is initializing > > > > In UI HMaster status shows like this *Assigning META region (since > 18mins, > > 39sec ago)* > > > > and i see the Hmaster logs are flowing with the following debug prints, > the > > log file is full of below prints, > > * > > 2012-08-22 01:08:19,637 DEBUG > > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > > Looked up root region location, > > > connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@49586cbd > ; > > serverName=hadoop-datanode1,60020,1345596463277 > > 2012-08-22 01:08:19,638 DEBUG > > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > > Looked up root region location, > > > connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@49586cbd > ; > > serverName=hadoop-datanode1,60020,1345596463277 > > 2012-08-22 01:08:19,639 DEBUG > > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > > Looked up root region location, > > > connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@49586cbd > ; > > serverName=hadoop-datanode1,60020,1345596463277* > > > > Please help me in debugging this. > > > > > > > > > > > > -- > > View this message in context: > http://apache-hbase.679495.n3.nabble.com/Problem-Bringing-up-the-HBase-cluster-tp4019948.html > > Sent from the HBase - Developer mailing list archive at Nabble.com. >
Re: Problem - Bringing up the HBase cluster
Hi, Please use the user mailing list (added at dest) for this type of questions instead of the dev list (now in bcc). It's a little bit strange to use the full distributed mode with a single region server. Is the Pseudo-distributed mode working? Check the number of datanodes vs. dfs.replication (default 3). If you have less datanodes then dfs.replication value, it won't work properly. Check as well that the region server is connected to the master. Cheers, On Wed, Aug 22, 2012 at 3:16 AM, kbmkumar wrote: > Hi, > I am trying to bring up a HBase cluster with 1 master and 1 one region > server. I am using > Hadoop 1.0.3 > Hbase 0.94.1 > > Starting the hdfs was straight forward and i could see the namenode up and > running successfully. But the problem is with Hbase. I followed all the > guidelines given in the Hbase cluster setup (fully distributed mode) and ran > the start-hbase.sh > > It started the Master, Region server and zookeeper (in the region server) as > per my configuration. But i am not sure the master is fully functional. When > i try to connect hbase shell and create table, it errors out saying > PleaseHoldException- Master is initializing > > In UI HMaster status shows like this *Assigning META region (since 18mins, > 39sec ago)* > > and i see the Hmaster logs are flowing with the following debug prints, the > log file is full of below prints, > * > 2012-08-22 01:08:19,637 DEBUG > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > Looked up root region location, > connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@49586cbd; > serverName=hadoop-datanode1,60020,1345596463277 > 2012-08-22 01:08:19,638 DEBUG > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > Looked up root region location, > connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@49586cbd; > serverName=hadoop-datanode1,60020,1345596463277 > 2012-08-22 01:08:19,639 DEBUG > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > Looked up root region location, > connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@49586cbd; > serverName=hadoop-datanode1,60020,1345596463277* > > Please help me in debugging this. > > > > > > -- > View this message in context: > http://apache-hbase.679495.n3.nabble.com/Problem-Bringing-up-the-HBase-cluster-tp4019948.html > Sent from the HBase - Developer mailing list archive at Nabble.com.
