testing co-processors

Hi all, I am looking for a way to test co-processors. Some thing similar to mrunit. is there a mock framework in hbase that I can use? thanks in advance. Sujee Maniyam (http://sujee.net)

utility to print (dump) HBase config (feedback appreciated)

Just wrote quick snippet to dump HBase config. snippet : https://github.com/sujee/print-hadoop-config/blob/master/src/main/java/config/HBaseConfig.java project : https://github.com/sujee/print-hadoop-config any feedback / suggestions appreciated thanks Sujee Maniyam (http://sujee.net)

Re: a utility to verify DNS on Hadoop/Hbase cluster

HI St.Ack Jira created and has patch/text for 'http://hbase.apache.org/book.html#dns' https://issues.apache.org/jira/browse/HBASE- please let me know if you need any other info to get this into the guide thanks in advance. cheers sujee

a utility to verify DNS on Hadoop/Hbase cluster

HI all, I was once stung by issue of DNS not working correctly on a Hadoop/Hbase cluster, it wasn't easy to debug. So I wrote a simple utility to verify DNS on a cluster (all machines) https://github.com/sujee/hadoop-dns-checker - It is written in pure Java; doesn't use any third party

Re: a utility to verify DNS on Hadoop/Hbase cluster

Nice Sujee.  Make a patch for the reference guide so folks can find it easy?  Add a footnote here: http://hbase.apache.org/book.html#dns? (If you write text into an issue, I'll take care of getting it into the guide).  Shouldn't we have this in hbase altogether? ./bin/hbase checkdns or

controlling zookeeper logging output

I have the following in my HBASE_HOME/conf/log4j.properites log4j.logger.org.apache.zookeeper=WARN log4j.logger.org.apache.hadoop.hbase=INFO log4j.logger.org.apache.hadoop.hbase.zookeeper=WARN still I see a lot of logging output from zookeeper from my hbase client. like: 12/01/16 11:31:07

Re: need feedback on PerformanceEvaluation with presplit option test code

with the JIRA here : https://issues.apache.org/jira/browse/HBASE-4440 and submit a patch. thanks very much Sujee http://sujee.net On Mon, Nov 28, 2011 at 12:35 PM, Sujee Maniyam su...@sujee.net wrote: Hi All I have added a presplit option to PerformanceEvaluation class.

need feedback on PerformanceEvaluation with presplit option test code

Hi All I have added a presplit option to PerformanceEvaluation class. I see the TestTable is created with splits. But when I run 'randomWrite' test (in MR mode) majority of the 'requests' are going to only one region server. Other region servers are busy as well, but catering to small number

storing MB sized files in HBase

HI All I have data files (binary) that are in 2-5 MB in size. Can I store them in a Hbase shell -- adjusting block-size and region-size ? or should I store them in HDFS and store the pointer in Hbase? http://wiki.apache.org/hadoop/Hbase/FAQ_Design#A3 says not to go beyond 10MB per cell. any

region size/count per regionserver

HI all, My HBase cluster is 10 nodes, each node has 12core , 48G RAM, 24TB disk, 10GEthernet. My region size is 1GB. Any guidelines on how many regions can a RS handle comfortably? I vaguely remember reading some where to have no more than 1000 regions / server; that comes to 1TB / server.

Re: region size/count per regionserver

optimizations for compactions in 0.92. In our case we have a pretty old setup and had way too many regions so we ran a few online merges to bring this down to like 80 regions/RS and it's working pretty well. J-D what is the region size you use? and is it 80 regions / table / region-server?

Re: tutorial : HBase performance testing

hehe J-D (hopefully first name!) I agree with your point that pre-splitting the table can make a big difference. Do the later versions of 'PerformanceEvaluation' class has an option to pre-split the table? I remember, when I ran this for the first time, only one region server is busy until

Re: question on HTablePool and threads

sounds like even I created an HTablePool and shared it among threads (which seems safe to do as pointed out here), I won't see much improvements for accessing the SAME table in multiple threads. correct? http://sujee.net

question on HTablePool and threads

with multiple threads. Is this a valid / thread-safe ? create HtablePool in 'main' pass HTablePool instance to threads each thread does a 'htablepool.get(table)' read / write to table 'htablepool.put (table) ' when done (all this is done within a single JVM) thanks Sujee Maniyam http

region splits, but not assigned to other region servers

Hi all I am loading data into a table using a standalone java client (multiple threads). I see the table splits into regions, but for a while the new regions are not distributed among region servers. (see screenshot, in this case all regions are in SN8 machine). So the updates go to one RS as

region servers not shutting down (v 0.90.1-cdh3u0, r)

this in the region server logs: 2011-06-10 12:03:55,940 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting on 70236052 2011-06-10 12:03:58,942 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting on 70236052 thanks very much for your help! Sujee Maniyam http://sujee.net

Re: region servers not shutting down (v 0.90.1-cdh3u0, r)

of the -ROOT- region? Is that so. Anything on why it won't go down earlier in the log? St.Ack On Fri, Jun 10, 2011 at 12:23 PM, Sujee Maniyam su...@sujee.net wrote: Hi all I am running Hbase on a 6 node cluster. HBase comes up fine, I can create a test table and put rows and scan. But I

Re: region servers not shutting down (v 0.90.1-cdh3u0, r)

://hbase.apache.org/book.html#dns BTW the clue comes from those kinda lines: 2011-06-10 12:03:50,975 INFO org.apache.hadoop.hbase.zookeeper.RegionServerTracker: No HServerInfo found for devperf-sn10.pcs.hds.com,60020,1307732557915 J-D On Fri, Jun 10, 2011 at 9:26 PM, Sujee Maniyam su

Re: HBase went down

Can you post some details about your cluster : number of machines, type of machines, how much data you are going through ..etc try increasing the zookeeper timeout in Hbase/conf/hbase-site.xml property namezookeeper.session.timeout/name value6/value /property

Re: dead-lock at HTable flusCommits with multiple clients...

more log : http://pastebin.com/nVYdJb3v HTable is not shared, each thread creates its own HTable instance. the import goes along for a few minutes. I can see 'requests' on HBase Master UI. Then the client hangs, requests become zero. http://sujee.net

dead-lock at HTable flusCommits with multiple clients...

I am importing data into Hbase with a client running 10 threads. I explicitly call 'flushCommit' from each thread (after a few thousand puts) Here is the thread-dump: pool-1-thread-20 prio=10 tid=0x41072800 nid=0x17d8 in Object.wait() [0x7fdaee6c8000] java.lang.Thread.State:

Re: dead-lock at HTable flusCommits with multiple clients...

forgot to mention, that I am using hbase 0.20.4