Using the Hadoop bundled in the lib directory of HBase

Hi, I'm new to HBase and have a stupid question about its dependency on Hadoop. Section 1.3.1.2 of (http://hbase.apache.org/notsoquick.html) says there is an "instance" of Hadoop in the lib directory of HBase. What exactly is meant by "instance"? Is it all I need, or do I need to get a "full

Re: HBase/Zookeeper -- System Fails when IP Address Changes

On Tue, Feb 1, 2011 at 7:16 AM, grailcattt wrote: > > I have hadoop/hbase running on a notebook as my dev env. I have everything > set up to use localhost which is defined as 127.0.0.1 in my /etc/hosts (and > removed other entries for localhost). > > The system works great all day, but when I go h

Re: Region server shutdown during writes (bad data nodes)

Please put up more from that log so we can see more around this failed region open. Can you check out the datanode to its side. Does it have errors? Is it 'peer' referred to below? (Usually there is the address who we are talking to). Pastebin it all. Thanks. St.Ack On Fri, Feb 4, 2011 at 4:1

hbase and hadoop capacity and load measurement

Hi, There: We have a hadoop/hbase cluster with 6 regionservers, double as task tracker and datanodes. They have 8G and 4x0.5T disk. I am using cdh3b2 distribution. I noticed that when the load is small, everything is happy. However, when we push enough data continuously to hbase and run map-

Region server shutdown during writes (bad data nodes)

Hello, We are running into a Region server shutdown again during write loads (90 clients) , with Connection rest by peer issue? Any suggestions.. Setup: 30 Nodes. Hbase 0.90.0, Hadoop-append , CentOS, dell 1950 6G RAM. 2011-02-04 02:36:16,808 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutp

Re: Region Servers Crashing during Random Reads

Under our load at su, the new gen would grow to max size and take 800+ ms. I would consider setting the ms goal to 20-40ms (what we get in prod now). At 1gb par new i would expect large pauses. Plus in my previous tests the promotion was like 75% even with a huge par new. This is all based on my b

Re: Type mismatch

I took Sujee's his example, which worked out of the box, and changed it to fit my problem. I am going line by line now, to see what he does differently. Mark On Fri, Feb 4, 2011 at 2:01 PM, Stack wrote: > (Thanks Sujee) > > What did you change in your src to get it going? > > St.Ack > > On Fri,

Re: Type mismatch

(Thanks Sujee) What did you change in your src to get it going? St.Ack On Fri, Feb 4, 2011 at 10:56 AM, Mark Kerzner wrote: > I found an example that works and uses the latest HBase API, > http://sujee.net/tech/articles/hbase-map-reduce-freq-counter/, you might > know about it, but for me it wa

Re: Type mismatch

I found an example that works and uses the latest HBase API, http://sujee.net/tech/articles/hbase-map-reduce-freq-counter/, you might know about it, but for me it was very helpful. Mark On Fri, Feb 4, 2011 at 11:55 AM, Stack wrote: > Its just an issue of matching your outputs to TOF. There ar

Re: NoServerforRegionexception

The below error is telling you that -ROOT- is not assigned. Grep logs to try figure what happened to it. Try restart to see if it brings it back. You are aware that pseudo-distributed mode is for dev and exploration only. It'll likely not do too well if you are trying to load a bunch of data in

Re: Type mismatch

Its just an issue of matching your outputs to TOF. There are examples of Reducer usage in the mapreduce package. They declare their types other than how you have it. See PutSortReducer and ImportTsv which uses it (and configures it up). St.Ack On Fri, Feb 4, 2011 at 7:08 AM, Mark Kerzner wro

Re: asynchronous hbase for batch inserts?

Checkout opentsdb. There are a bunch of examples in the code base of it using asynchbase. Grep HBaseClient or 'client' in the opentsdb database. St.Ack On Fri, Feb 4, 2011 at 4:43 AM, Shuja Rehman wrote: > Hi > > I was wondering if anyone can share the working example for batch inserting > usin

Re: Region Servers Crashing during Random Reads

On Fri, Feb 4, 2011 at 12:20 AM, Lars George wrote: > I saw the -XX:MaxGCPauseMillis option too and assumed it is not that > effective as it was never suggested so far. So it was simply not tried > yet and someone has to be the guinea pig? > Yeah, haven't had good experience with these upper-boun

Re: Queries regarding REST API

I mean support for the text/plain content-type with a similar behavior as application/octet-stream, except rather than providing 8 bit data all non-ASCII characters will be suitably escaped. To do so for the XML and JSON cases, the way Jersey works we'd have to convert all byte[] to strings wit

Re: Type mismatch

I tried 0.90 - same error. I am going to try to build HBase from code and include this code in my debugging session, to step through it. But I must be doing something wrong. How does one write to HBase in the Reducer, is there any example!??? Thank you! Mark On Fri, Feb 4, 2011 at 12:38 AM, Sta

Re: is there any tool that facilitate the import of data to hbase

I am looking forward to it :) btw, I found this informative video about Sqoop. http://www.cloudera.com/blog/2009/12/hadoop-world-sqoop-database-import-for-hadoop/ On Thu, Feb 3, 2011 at 4:44 PM, arv...@cloudera.com wrote: > On Thu, Feb 3, 2011 at 2:17 PM, Weishung Chung wrote: > > > that's aweso

Re: question about HTableDescriptor

That's really good explanation! Thanks alot :D Have a good day ! On Fri, Feb 4, 2011 at 2:27 AM, Lars George wrote: > You can see this from the first few lines in the HTableDescriptor.java, > i.e. > > public static final String FAMILIES = "FAMILIES"; > public static final ImmutableBytesWritabl

Re: NoServerforRegionexception

Hi, I am having the Hadoop-CDH3 pseudo distributed environment. I am running the Map Reduce program to do the Bulk load of data into the Hbase-0.89, I am getting the following exception. org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region > � � � �at

asynchronous hbase for batch inserts?

Hi I was wondering if anyone can share the working example for batch inserting using following asynchronous hbase. http://tsunanet.net/~tsuna/asynchbase/api/ More specific can provide example equivalent to this for(i=0; ihttp://pk.linkedin.com/in/sh

Re: question about HTableDescriptor

You can see this from the first few lines in the HTableDescriptor.java, i.e. public static final String FAMILIES = "FAMILIES"; public static final ImmutableBytesWritable FAMILIES_KEY = new ImmutableBytesWritable(Bytes.toBytes(FAMILIES)); public static final String MAX_FILESIZE = "MAX_FIL

Re: Region Servers Crashing during Random Reads

I saw the -XX:MaxGCPauseMillis option too and assumed it is not that effective as it was never suggested so far. So it was simply not tried yet and someone has to be the guinea pig? So that 6M NewSize in the Wiki was OK previously and now 256M is actually better? The represent opposite ideas and I

Re: question about HTableDescriptor

Thank you but I would like to know what kinds of key value pairs that serve as the table descriptors in the values map. I know the column map stores the column name & column descriptor pair as the map entry. I should spend more time on the codes again tomorrow. Sent from my iPhone On Feb 4

Re: Region Servers Crashing during Random Reads

Did some reading through the hotspot source to look into this. In the past I've advised small newsize to keep the pause times down on the minor collections, but after looking at the source I think there's a better way: -XX:MaxGCPauseMillis=100 (or whatever pause seems reasonable to you). If you do

Re: question about HTableDescriptor

Hi, Did you read the comment above? /** * Private constructor used internally creating table descriptors for * catalog tables: e.g. .META. and -ROOT-. */ Explains it, no? Lars On Fri, Feb 4, 2011 at 8:41 AM, Weishung Chung wrote: > I am looking at the following protected HTableDesc