unable to start regionservers on slaves in hbase
i have a problem starting regionservers on slave pc,s. when i enlist only master pc in conf/regionservers every thing works fine but when i add two more slaves to it the hbase does not start . if i delete all hbase folders in the tmp folder from all pc,s and then start regionserver (with 3 regionservers enlisted)the hbase gets started but when i try to create a table it again fails(gets stuck) pls anyone help i am using hadoop 0.20.0 which is working fine and hbase 0.92.0 i have 3 pc's in cluster(all running windows xp) one master and two slaves also tell that is DNS (both forward and backward lookup working)necessary for hbase in my case is there any way to replicate hbase table to all region servers i.e. i want to have a copy of table at each pc and want to access them locally(when i execute map task they should use their local copy of hbase table) plz help..!! thanx in advance -- View this message in context: http://old.nabble.com/unable-to-start-regionservers-on-slaves-in-hbase-tp33673626p33673626.html Sent from the HBase User mailing list archive at Nabble.com.
RE: Not a host:port issue
Thanks Stack, We have successfully restored our Hadoop/HBase cluster to a healthy state. It seems that moving the pig and zookeeper references to the back of the HADOOP_CLASSPATH so that the backticked `hbase classpath` came first in the HADOOP_CLASSPATH resolved the issue. I have not confirmed this, but I have suspicions that the old hbase client references were coming from the pig contributions to the classpath. I am wondering if perhaps the current hbase and zookeeper libs would have been included in the pig jar when we built from source. So that the old hbase client would be found inside the pig jar at the front of the classpath in the past...? WRT to the DiskErrorException we mentioned in the previous email, that was being caused by one of the key worker nodes having no disk space left. After deleting a few redundant files from each of our cluster nodes we were back up and running. Thanks for all your help. Tom and Royston -Original Message- From: saint@gmail.com [mailto:saint@gmail.com] On Behalf Of Stack Sent: 12 April 2012 05:14 To: user@hbase.apache.org Subject: Re: Not a host:port issue On Wed, Apr 11, 2012 at 5:14 AM, Tom Wilcox wrote: > 1) Removed all references to HADOOP_CLASSPATH in hadoop-env.sh and replaced > with the following so that any initial HADOOP_CLASSPATH settings have > precedence: > > # Extra Java CLASSPATH elements. Optional. > export HADOOP_CLASSPATH="$HADOOP_CLASSPATH:$ZOOKEEPER_INSTALL/*" > export HADOOP_CLASSPATH="$HADOOP_CLASSPATH:$PIGDIR/*" > Above you are including a version that is probably different from hbase's and its being stuck ahead of ours on the classpath IIRC. Not sure why this would give you the behavior you are seeing. I'd have thought it'd have made no difference. Could it be that your hbase is homed at different locations up in zk and you are picking up an old home because you are picking up an old config? (It doesn't looks so when I look at your pastebins -- you seem to have same ensemble in each case w/ same /zookeeper_data homedir). Different zk instances up for each test? I'm a little baffled. > 2) Ran the job with the following (so that HADOOP_CLASSPATH contained all > appropriate HBase API jars): > > HADOOP_CLASSPATH=`hbase classpath` hadoop jar SampleUploader.jar > uk.org.cse.ingestion.SampleUploader sample.10.csv tomstable dat no > > We are now dealing with the following error: > > [sshexec] org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not > find any valid local directory for > taskTracker/hadoop1/distcache/-6735763131868259398_188156722_559071878/namenode/tmp/mapred/staging/hadoop1/.staging/job_201204111219_0013/libjars/hbase-0.95-SNAPSHOT.jar > [sshexec] at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:381) > [sshexec] at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:146) > [sshexec] at > org.apache.hadoop.filecache.TrackerDistributedCacheManager.getLocalCache(TrackerDistributedCacheManager.java:172) > [sshexec] at > org.apache.hadoop.filecache.TaskDistributedCacheManager.setupCache(TaskDistributedCacheManager.java:187) > [sshexec] at > org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1212) > [sshexec] at java.security.AccessController.doPrivileged(Native Method) > [sshexec] at javax.security.auth.Subject.doAs(Subject.java:396) > [sshexec] at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093) > [sshexec] at > org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1203) > [sshexec] at > org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1118) > [sshexec] at > org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2430) > [sshexec] at java.lang.Thread.run(Thread.java:662) > [sshexec] > These dirs set up out on your cluster? Google it. There's a couple of possible explanations. You might go review how to package a jar for mapreduce. It can be a little tricky to get right. Best to ship in the job jar all of its dependencies and keep your cluster CLASSPATH clean. See the trick where the hbase mapreduce jobs pull in jars of the CLASSPATH that its needs down in TableMapReduceUtil#addDependencyJars. Perhaps review too the hbase story on mapreduce and CLASSPATHing: http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/package-summary.html#classpath Good luck lads, St.Ack
Re: Performance Optimization techniques HBase
In case you haven't checked yet: * http://hbase.apache.org/bulk-loads.html * http://hbase.apache.org/book.html Alex Baranau -- Sematext :: http://blog.sematext.com/ :: Solr - Lucene - Hadoop - HBase On Wed, Apr 11, 2012 at 10:06 PM, Neha wrote: > I am a newbie in HBase. I am working on importing large scale data. > What are the general optimization techniques/ways that should be used, > while > developing such a code? > We are using HBase Java API (Mapper/Reducer architecture). > Also, we don't want to do the alterations at byte level, using HFiles (as > done > by ImportTsv). > Could someone please suggest some techniques/ways to do this efficiently. > >
Re: Performance Optimization techniques HBase
And the "HBase: THe Definitive Guide" book from Lars George http://shop.oreilly.com/product/0636920014348.do On 04/12/2012 08:14 AM, Alex Baranau wrote: In case you haven't checked yet: * http://hbase.apache.org/bulk-loads.html * http://hbase.apache.org/book.html Alex Baranau -- Sematext :: http://blog.sematext.com/ :: Solr - Lucene - Hadoop - HBase On Wed, Apr 11, 2012 at 10:06 PM, Neha wrote: I am a newbie in HBase. I am working on importing large scale data. What are the general optimization techniques/ways that should be used, while developing such a code? We are using HBase Java API (Mapper/Reducer architecture). Also, we don't want to do the alterations at byte level, using HFiles (as done by ImportTsv). Could someone please suggest some techniques/ways to do this efficiently. 10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS... CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION http://www.uci.cu http://www.facebook.com/universidad.uci http://www.flickr.com/photos/universidad_uci -- Marcos Luis Ortíz Valmaseda (@marcosluis2186) Data Engineer at UCI http://marcosluis2186.posterous.com 10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS... CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION http://www.uci.cu http://www.facebook.com/universidad.uci http://www.flickr.com/photos/universidad_uci
Re: Not a host:port issue
On Thu, Apr 12, 2012 at 2:52 AM, Tom Wilcox wrote: > I have not confirmed this, but I have suspicions that the old hbase client > references were coming from the pig contributions to the classpath. I am > wondering if perhaps the current hbase and zookeeper libs would have been > included in the pig jar when we built from source. So that the old hbase > client would be found inside the pig jar at the front of the classpath in the > past...? > The joys of classpath! > WRT to the DiskErrorException we mentioned in the previous email, that was > being caused by one of the key worker nodes having no disk space left. After > deleting a few redundant files from each of our cluster nodes we were back up > and running. > Good to hear you are up again. St.Ack
Re: unable to start regionservers on slaves in hbase
On Thu, Apr 12, 2012 at 12:50 AM, amsal wrote: > > i have a problem starting regionservers on slave pc,s. when i enlist only > master pc in conf/regionservers every thing works fine but when i add two > more slaves to it the hbase does not start . if i delete all hbase > folders in the tmp folder from all pc,s and then start regionserver (with 3 > regionservers enlisted)the hbase gets started but when i try to create a > table it again fails(gets stuck) pls anyone help i am using hadoop > 0.20.0 which is working fine and hbase 0.92.0 i have 3 pc's in cluster(all > running windows xp) one master and two slaves > HBase will not work w/ hadoop 0.20.0. See hbase requirements: http://hbase.apache.org/book.html#configuration What are the errors you are seeing? > also tell that is DNS (both forward and backward lookup working)necessary > for hbase in my case is there any way to replicate hbase table to all > region servers i.e. i want to have a copy of table at each pc and want to > access them locally(when i execute map task they should use their local copy > of hbase table) plz help..!! thanx in advance > Thats not how it works. See the reference guide cited above. St.Ack
Re: Schema Updates: what do you do today?
Still doing it old school unless I have to alter a bunch of tables for
the same reason, like enabling replication.
J-D
On Mon, Apr 9, 2012 at 9:08 AM, Ian Varley wrote:
> All:
>
> I'm doing a little research into various ways to apply schema modifications
> to an HBase cluster. Anybody care to share with the list what you currently
> do? E.g.
>
> - Connect via the HBase shell and manually issue commands ("create",
> "disable", "alter", etc.)
> - Write one-off scripts that do the above
> - Write tools that read from a static schema definition and then apply
> changes to a cluster (e.g. using HBaseAdmin)
>
> etc. My supposition is that some additional tooling in this area, to
> consolidate stuff everybody already does on their own, might be helpful. In
> light of recent discussions on the dev list about various ways to alter the
> schema on a running cluster, it seems like this area is still a bit of a
> "wild west" in the HBase community, both in how HBase works and in what
> people do in practice.
>
> What do you do today for schema changes, and what would you like to do in an
> ideal world?
>
> Thanks,
> Ian
Re: Not a host:port issue
On 12 Apr 2012, at 15:49, Stack wrote: > On Thu, Apr 12, 2012 at 2:52 AM, Tom Wilcox wrote: >> I have not confirmed this, but I have suspicions that the old hbase client >> references were coming from the pig contributions to the classpath. I am >> wondering if perhaps the current hbase and zookeeper libs would have been >> included in the pig jar when we built from source. So that the old hbase >> client would be found inside the pig jar at the front of the classpath in >> the past...? >> > > The joys of classpath! Amen to that! You think you've made friends with classpath then it whacks you round the back of the head. Royston
Re: Is HBase Thread-Safety?
There only 'transaction' HBase provides is row-level transaction. On Thu, Apr 12, 2012 at 3:10 PM, Bing Li wrote: > Dear all, > > Is HBase thread-safety? Do I need to consider the consistency issue when > manipulating HBase? > > Thanks so much! > > Best regards, > Bing > -- --Sean
Re: Is HBase Thread-Safety?
re: "Is HBase thread-safety?" HTable instances are not thread safe, though. http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html On 4/12/12 6:10 PM, "Bing Li" wrote: >Dear all, > >Is HBase thread-safety? Do I need to consider the consistency issue when >manipulating HBase? > >Thanks so much! > >Best regards, >Bing
Re: Is HBase Thread-Safety?
Hi Bing, Which part? The server certainly is thread safe. The client is not, at least not all the way through. The main consideration is HTable, which is not thread safe, you need to create one instance for each thread (HBASE-4805 makes that much cheaper), store the HTable in a ThreadLocal after creation, or use HTablePool. Please let me know if that answers your question. Thanks. -- Lars - Original Message - From: Bing Li To: hbase-u...@hadoop.apache.org; user Cc: Sent: Thursday, April 12, 2012 3:10 PM Subject: Is HBase Thread-Safety? Dear all, Is HBase thread-safety? Do I need to consider the consistency issue when manipulating HBase? Thanks so much! Best regards, Bing
Re: Is HBase Thread-Safety?
Dear Iars, Thanks so much for your reply! In my case, I need to overwrite or update a HTable. If reading during the process of updating or overwriting, any exceptions will be thrown by HBase? If multiple instances for a HTable are used by multiple threads, there must be inconsistency among them, right? I guess caching must be done in HBase. So retrieving in HTable must be almost as fast as in memory? Best regards, Bing On Fri, Apr 13, 2012 at 6:17 AM, lars hofhansl wrote: > Hi Bing, > > Which part? The server certainly is thread safe. > The client is not, at least not all the way through. > > The main consideration is HTable, which is not thread safe, you need to > create one instance for each thread > (HBASE-4805 makes that much cheaper), store the HTable in a ThreadLocal > after creation, or use HTablePool. > > Please let me know if that answers your question. > > Thanks. > > -- Lars > > > - Original Message - > From: Bing Li > To: hbase-u...@hadoop.apache.org; user > Cc: > Sent: Thursday, April 12, 2012 3:10 PM > Subject: Is HBase Thread-Safety? > > Dear all, > > Is HBase thread-safety? Do I need to consider the consistency issue when > manipulating HBase? > > Thanks so much! > > Best regards, > Bing > >
Zookeeper available but no active master location found
"client.HConnectionManager$HConnectionImplementation: ZooKeeper available but no active master location found" Having a problem with master startup that I have not seen before. running the following packages: hadoop-hbase-0.90.4+49.137-1 hadoop-0.20-secondarynamenode-0.20.2+923.197-1 hadoop-hbase-thrift-0.90.4+49.137-1 hadoop-zookeeper-3.3.4+19.3-1 hadoop-0.20-datanode-0.20.2+923.197-1 hadoop-0.20-namenode-0.20.2+923.197-1 hadoop-0.20-tasktracker-0.20.2+923.197-1 hadoop-hbase-regionserver-0.90.4+49.137-1 hadoop-zookeeper-server-3.3.4+19.3-1 hadoop-0.20-0.20.2+923.197-1 hadoop-0.20-jobtracker-0.20.2+923.197-1 hadoop-hbase-master-0.90.4+49.137-1 [root@ip-10-251-27-130 logs]# java -version java version "1.6.0_31" Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode) I start master and region server on another node. Master is initialized, but as soon as I try to check the master_status or do a zkdump via web interface, it blows up with: 2012-04-12 19:16:10,453 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: ZooKeeper available but no active master location found 2012-04-12 19:16:10,453 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: getMaster attempt 10 of 10 failed; retrying after sleep of 16000 I am running three zookeepers: # The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit=10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. dataDir=/mnt/zookeeper # The maximum number of zookeeper client connections maxClientCnxns=2000 # the port at which the clients will connect clientPort=2181 server.1=10.251.27.130:2888:3888 server.2=10.250.9.220:2888:3888 server.3=10.251.110.50:2888:3888 I can telnet to the zookeepers just fine. Here is my hbase-site.xml file: hbase.rootdir hdfs://namenode:9000/hbase hbase.cluster.distributed true hbase.zookeeper.quorum 10.251.27.130,10.250.9.220,10.251.110.50 hbase.zookeeper.property.dataDir /hadoop/zookeeper/data hbase.zookeeper.property.maxClientCnxns 2000 true Any thoughts? Any help is greatly appreciated. Thanks Henri Pipe
Re: Is HBase Thread-Safety?
1. A pre-row lock is here during the update, so other clients will block whild on client performs an update.(see HRegion.put 's annotaion), no exception. In the client side, while a process is updating, it may not reach the buffersize so the other process may read the original value, I think. 2. What kind of inconsistency? different value on the same row's qualifier? 3.I don't know the truely realize in code. There Is caching, but everytime you call methods like Htable.get, it still need connect to server to retrieve data——so, not as fast as in memory, isn't it? Best regards, nn 2012/4/13 Bing Li > Dear Iars, > > Thanks so much for your reply! > > In my case, I need to overwrite or update a HTable. If reading during the > process of updating or overwriting, any exceptions will be thrown by HBase? > > If multiple instances for a HTable are used by multiple threads, there must > be inconsistency among them, right? > > I guess caching must be done in HBase. So retrieving in HTable must be > almost as fast as in memory? > > Best regards, > Bing > > On Fri, Apr 13, 2012 at 6:17 AM, lars hofhansl > wrote: > > > Hi Bing, > > > > Which part? The server certainly is thread safe. > > The client is not, at least not all the way through. > > > > The main consideration is HTable, which is not thread safe, you need to > > create one instance for each thread > > (HBASE-4805 makes that much cheaper), store the HTable in a ThreadLocal > > after creation, or use HTablePool. > > > > Please let me know if that answers your question. > > > > Thanks. > > > > -- Lars > > > > > > - Original Message - > > From: Bing Li > > To: hbase-u...@hadoop.apache.org; user > > Cc: > > Sent: Thursday, April 12, 2012 3:10 PM > > Subject: Is HBase Thread-Safety? > > > > Dear all, > > > > Is HBase thread-safety? Do I need to consider the consistency issue when > > manipulating HBase? > > > > Thanks so much! > > > > Best regards, > > Bing > > > > >
