Re: Need help in running rowcounter mapreduce job in Hadoop cluster

2012-06-12 Thread AnandaVelMurugan Chandra Mohan
Thanks for quick response That was a silly mistake. Sorry about that. But now I get the following error Exception in thread "main" java.lang.NoClassDefFoundError: com/google/common/collect/Multimap at org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:43) at sun.reflect.Nat

RE: Need help in running rowcounter mapreduce job in Hadoop cluster

2012-06-12 Thread Gopinathan A
Use ./hadoop jar /usr/local/hbase-0.92.1-security/hbase-0.92.1-security.jar rowcounter Thanks & Regards, Gopinathan A *** This e-mail and attachments contain confidential information from HUAWEI, which is intend

Re: one HRegionServer means one DataNode?

2012-06-12 Thread AnExplorer
Thanks Doug. I have gone through this and theoretically kind of understood it. But after setting up the cluster, I was not able to make out that where is RegionServer and where I can find that. Would really seek help in future from you guys.Thanks a lot!! Doug Meil-2 wrote: > > > Hi there- >

RE: one HRegionServer means one DataNode?

2012-06-12 Thread AnExplorer
Thanks Ram!! So, there are 3 datanodes in the cluster. Is there any way to find out that whether these datanodes are served by one RegionServer or more? where can we see that? Thanks again.. Ramkrishna.S.Vasudevan wrote: > > See response inline. > >> -Original Message- >> From: AnExplo

Re: HBase first steps: Design a table

2012-06-12 Thread Jean-Marc Spaggiari
Oh! Nice! I really like the secondary index approach. A bit manual, but at least I can tweak it the way I want... Thanks for the reply! Seems that I will have to read again the section about the timestamp because it might be very helpful for me. Regarding the key, for me it's a MD5 of a long str

Re: HBase first steps: Design a table

2012-06-12 Thread Jean-Marc Spaggiari
Thanks Alan. I was not sure what "row1" on the example was for... So I will go that way. And the example is perfect this it will be done in Java too. Usually I'm inserting about 40 000 rows at a time. Should I do 40 000 calls to put? Or is there any "bulkinsert" method? 2012/6/12, Alan Chaney :

Re: HBase first steps: Design a table

2012-06-12 Thread Damien HARDY
Hi Jean-Marc, I reply in your text. Le 12/06/2012 23:42, Jean-Marc Spaggiari a écrit : > Hi, > > I have read all the documentation here > http://hbase.apache.org/book/book.html and I now have few questions. > > I currently have a mysql table with millions of lines (4 for now, but > it's growing b

Re: HBase first steps: Design a table

2012-06-12 Thread Alan Chaney
On 6/12/2012 2:42 PM, Jean-Marc Spaggiari wrote: Here is what the table looks like: +--+--+--+-+-+---+ | Field| Type | Null | Key | Default | Extra | +--+--+--+-+-+---+ | IDLow| bigint(

HBase first steps: Design a table

2012-06-12 Thread Jean-Marc Spaggiari
Hi, I have read all the documentation here http://hbase.apache.org/book/book.html and I now have few questions. I currently have a mysql table with millions of lines (4 for now, but it's growing by 4 millions a month). It's running on a fast computer, but it's still way to slow when it's time to

Re: Pre-split table using shell

2012-06-12 Thread Michael Segel
?Inferred sigh of despair? Was it that obvious? :-) I'm not sure what hardware you're running on so its hard to say. Here's the problem... On each DN, you're running a DN and a RS. Assuming that you're not going to run a TT or do any M/R to push/pull data in and out of HBase. You don't have

RE: Master stuck trying to rebalance regions to a zombie region servers.

2012-06-12 Thread Pablo Musa
Hey guys, I had a similar problem a week ago and could not solve the problem in a clean way. I restarted the master and region servers so everything could work again. As Ram said, I think the best solution is to update HBase, which is not very easy but still the best option we found. Regards, P

Re: Pre-split table using shell

2012-06-12 Thread Simon Kelly
Using the API to create the splits worked. The data is now evenly spread across all the regions. However every time I tried to create a table the HBase master crashed. I used the class listed here http://pastebin.com/i1yFVEwj as follows: ./hbase CreateTable The table gets created but HBase maste

Re: one HRegionServer means one DataNode?

2012-06-12 Thread Christian Schäfer
As one picture could give more than thousand words...check out the first figure on  http://www.larsgeorge.com/search?q=HBase+Architecture It's the best figure I know to understand how Hadoop(here HDFS) and HBase are related to each other. Von: AnExplorer

Re: Pre-split table using shell

2012-06-12 Thread Simon Kelly
No, this isn't on EC2 and yes, its (supposed to be) production. Please elaboration on your inferred sigh of dispair On 12 June 2012 15:48, Michael Segel wrote: > Ok... > > Please tell me that this isn't a production system. > > Is this on EC2? > > On Jun 12, 2012, at 6:55 AM, Simon Kelly wro

Re: Pre-split table using shell

2012-06-12 Thread Michael Segel
Ok... Please tell me that this isn't a production system. Is this on EC2? On Jun 12, 2012, at 6:55 AM, Simon Kelly wrote: > Thanks Michael > > I'm 100% sure its not the UUID distribution that's causing the problem. I'm > going to try us the API to create the table and see if that changes thi

Re: Getting started

2012-06-12 Thread Jean-Marc Spaggiari
Hi Shashwat, I just figured that my issue already got addressed on this list: http://www.mail-archive.com/user@hbase.apache.org/msg12722.html It seems to be related to IPV6 configuration. So I disabled it. Regarding SSH, it's working passwordless as requested with both localhost and the host name

Re: Getting started

2012-06-12 Thread shashwat shriparv
Did you configure ssh?? if yes check if you can ssh what is your computer name you can get it using hostname at terminal. it should give you the name you are specifying the hosts file. On Tue, Jun 12, 2012 at 5:57 PM, Jean-Marc Spaggiari < jean-m...@spaggiari.org> wrote: > Hi, > > I'm new on HB

Re: one HRegionServer means one DataNode?

2012-06-12 Thread Doug Meil
Hi there- You probably should at least read this part of the RefGuide for an introŠ http://hbase.apache.org/book.html#architecture Š welcome to HBase! On 6/12/12 5:48 AM, "AnExplorer" wrote: > >Hi..I am new to Hbase. Can anyone please suggest that one HRegionServer >means >one DataNode? C

Getting started

2012-06-12 Thread Jean-Marc Spaggiari
Hi, I'm new on HBase and I'm trying to follow the "Quick Start" steps, but I'm facing some issues. I read a bit more of the documentation, and tried to update my hosts file but it's still not working. It seems the server is started, but it's not able to reach himself. Here is my hostfile: phen

Re: Pre-split table using shell

2012-06-12 Thread Simon Kelly
Thanks Michael I'm 100% sure its not the UUID distribution that's causing the problem. I'm going to try us the API to create the table and see if that changes things. The reason I want to pre-split the table is that HBase doesn't handle the initial load to a single regionserver and I can't start

RE: one HRegionServer means one DataNode?

2012-06-12 Thread Ramkrishna.S.Vasudevan
See response inline. > -Original Message- > From: AnExplorer [mailto:satishtha...@gmail.com] > Sent: Tuesday, June 12, 2012 5:05 PM > To: hbase-u...@hadoop.apache.org > Subject: RE: one HRegionServer means one DataNode? > > > Thanks for the response Ram!! It is pretty much clear to me no

Re: Pre-split table using shell

2012-06-12 Thread Michael Segel
Ok, Now that I'm awake, and am drinking my first cup of joe... If you just generate UUIDs you are not going to have an even distribution. Nor are they going to be truly random due to how the machines are generating their random numbers. But this is not important in solving your problem Th

RE: one HRegionServer means one DataNode?

2012-06-12 Thread AnExplorer
Thanks for the response Ram!! It is pretty much clear to me now that RegionServers are part of HBase and Datanodes are part of HDFS. Actually I have a query - I had setup a hadoop cluster with one master and 3 slaves. Are these three slaves called RegionServer? And if yes How many data nodes are t

Re: Pre-split table using shell

2012-06-12 Thread Oliver Meyn (GBIF)
Hi Simon, I might be wrong but I'm pretty sure the splits file you specify is assumed to be full of strings. So even though they look like bytes they're being interpreted as the string value (like '\x00') instead of the actual byte \x00. The only way I could get the byte representation of int

RE: Master stuck trying to rebalance regions to a zombie region servers.

2012-06-12 Thread Ramkrishna.S.Vasudevan
Hi Pradeep, Many changes have happened from the version that you are specifying upto the recent version. So may be you can try out latest versions. Regards Ram > -Original Message- > From: Pradeep Gopaluni [mailto:pradeep.gopal...@gmail.com] > Sent: Tuesday, June 12, 2012 4:35 PM > To:

Master stuck trying to rebalance regions to a zombie region servers.

2012-06-12 Thread Pradeep Gopaluni
Hi, We are using HBASE 0.90.3 and Hadoop 0.20.205 for our cluster. Today, we have observed that the HBASE master was stuck because of zombie regionservers that was responding to ping but was stuck. The logs show that the master tried to re-assign its regions from an overloaded (ServerA) and one

RE: one HRegionServer means one DataNode?

2012-06-12 Thread Ramkrishna.S.Vasudevan
Hi Region Server is not DataNode. DataNodes are part of HDFS. RegionServers are part of HBase. HBase uses HDFS to store data and in the process of storing data DataNodes are used by HDFS. DataNodes are the actual machines where the data resides. Regards Ram > -Original Message- > F

one HRegionServer means one DataNode?

2012-06-12 Thread AnExplorer
Hi..I am new to Hbase. Can anyone please suggest that one HRegionServer means one DataNode? Can there be multiple data nodes in one HRegionServer??:confused: -- View this message in context: http://old.nabble.com/one-HRegionServer-means-one-DataNode--tp33998720p33998720.html Sent from the HBase

Repairing a table

2012-06-12 Thread Mario Lassnig
Hello, I'm a bit stuck repairing a faulty table (on Hbase 0.92.1-cdh4.0.0, Hadoop 2.0.0-cdh4.0.0) There is a region in transition that doesn't finish: Region State bf2025f4bc154914b5942af4e72ea063 counter_traces,1329773878.35_766a0b4df75e4381a686fbc07db9e333,1339425291230.bf2025f4bc154914b5

Re: Pre-split table using shell

2012-06-12 Thread Simon Kelly
Yes, I'm aware that UUID's are designed to be unique and not evenly distributed but I wouldn't expect a big gap in their distribution either. The other thing that is really confusing me is that the regions splits aren't lexicographical sorted. Perhaps there is a problem with the way I'm specifying

Re: Pre-split table using shell

2012-06-12 Thread Michael Segel
UUIDs are unique but not necessarily random and even in random samplings, you may not see an even distribution except over time. Sent from my iPhone On Jun 12, 2012, at 3:18 AM, "Simon Kelly" wrote: > Hi > > I'm getting some unexpected results with a pre-split table where some of > the regio