Re: RowLocks

[Kristoffer Sjögren]

Maybe I should illustrate with a specific usecase.

I want to create a unique row with a rowkey that look like this:
id+timestamp

The timestamp (in millis) is provided by the user. It is ONLY the id that
dictate uniqueness, not the timestamp. So there is a race condition here.

My reasoning is as follows.

1) Lock the id (without the timestamp).

2.1) If the lock was aquired. Scan for the row.
2.2) If the row was not found, create it and set a counter on the
id+timestamp row to 0.
2.3) If the row is found, increase a counter on the id+timestamp row.
2.4) Release the lock.

If the lock was NOT aquired.

3.1) Not really sure how to proceed here. The easiest way is probably to
wait until the lock is released (like a SELECT FOR UPDATE). Retries would
also work.

It is very important that we do not loose the increments or create multiple
rows with same id with different timestamps when there are race conditions.



On Thu, Aug 29, 2013 at 6:22 AM, lars hofhansl  wrote:

> Specifically the API has been removed because it had never actually worked
> correctly.
>
>
> Rowlocks are used by RegionServers for intra-region operations.
> As such they are ephemeral, in-memory constructs, that cannot reliably
> outlive a single RPC request.
> The HTable rowlock API allowed you to create a rowlocks and hold it over
> multiple RPCs, which would break if f.e. a region is moved or split.
>
> -- Lars
> 
> From: Ted Yu 
> To: "user@hbase.apache.org" 
> Sent: Wednesday, August 28, 2013 8:01 PM
> Subject: Re: RowLocks
>
>
> The API is no longer a public API
>
> Thanks
>
>
> On Wed, Aug 28, 2013 at 7:58 PM, Michael Segel  >wrote:
>
> > Ted,
> > Can you clarify...
> > Do you mean the API is no longer a public API, or do you mean no more RLL
> > for atomic writes?
> >
> > On Aug 28, 2013, at 5:18 PM, Ted Yu  wrote:
> >
> > > RowLock API has been removed in 0.96.
> > >
> > > Can you tell us your use case ?
> > >
> > >
> > > On Wed, Aug 28, 2013 at 3:14 PM, Kristoffer Sjögren  > >wrote:
> > >
> > >> Hi
> > >>
> > >> About the internals of locking a row in hbase.
> > >>
> > >> Does hbase row locks map one-to-one with a locks in zookeeper or are
> > there
> > >> any optimizations based on the fact that a row only exist on a single
> > >> machine?
> > >>
> > >> Cheers,
> > >> -Kristoffer
> > >>
> >
> > The opinions expressed here are mine, while they may reflect a cognitive
> > thought, that is purely accidental.
> > Use at your own risk.
> > Michael Segel
> > michael_segel (AT) hotmail.com
> >
> >
> >
> >
> >
> >
>

Re: experiencing high latency for few reads in HBase

[lars hofhansl]

A 1s SLA is tough in HBase (or any large memory JVM application).


Maybe, if you presplit your table, play with JDK7 and the G1 collector, but 
nobody here will vouch for such an SLA in the 99th percentile.
I heard some folks have experimented with 30GB heaps and G1 and have reported 
max GC times of 200ms, but I have not verified that.

-- Lars



- Original Message -
From: Saurabh Yahoo 
To: "user@hbase.apache.org" 
Cc: "user@hbase.apache.org" 
Sent: Wednesday, August 28, 2013 3:17 PM
Subject: Re: experiencing high latency for few reads in HBase 

Hi Vlad,

Thanks for your response.

1. Our SLA is less than one sec. we cannot afford latency more than 1 sec. 

We can increase heap size if that help, we have enough memory on server. What 
would be the optimal heap size? 

2. Cache hit ratio is 95%.  One thing I don't understand that we have allocated 
only 4gb for block cache out of 12gb. That left 8gb for rest of JVM. There is 
no write. Memcache is empty. Is 8gb not enough for hbase to process the 
requests? What are the most memory consuming objects in region server? 

3. We will change the cf to IN_memory and report back performance difference.  

Thanks,
Saurabh. 

On Aug 28, 2013, at 3:15 PM, Vladimir Rodionov  wrote:

> 1. 4 sec max latency is not that bad taking into account 12GB heap.  It can 
> be much larger. What is your SLA?
> 2. Block evictions is the result of a poor cache hit rate and the root cause 
> of a periodic stop-the-world GC pauses (max latencies
>    latencies you have been observing in the test)
> 3. Block cache consists of 3 parts (25% young generation, 50% - tenured, 25% 
> - permanent). Permanent part is for CF with
> IN_MEMORY = true (you can specify this when you create CF).  Block first 
> stored in 'young gen' space, then gets promoted to 'tenured gen' space
> (or gets evicted). May be your 'perm gen' space is underutilized? This is 
> exact 25% of 4GB (1GB). Although HBase LruBlockCache should use all the space 
> allocated for block cache -
> there is no guarantee (as usual). If you don have in_memory column families 
> you may decrease
> 
> 
> 
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodio...@carrieriq.com
> 
> 
> From: Saurabh Yahoo [saurabh...@yahoo.com]
> Sent: Wednesday, August 28, 2013 5:10 AM
> To: user@hbase.apache.org
> Subject: experiencing high latency for few reads in HBase
> 
> Hi,
> 
> We are running a stress test in our 5 node cluster and we are getting the 
> expected mean latency of 10ms. But we are seeing around 20 reads out of 25 
> million reads having latency more than 4 seconds. Can anyone provide the 
> insight what we can do to meet below second SLA for each and every read?
> 
> We observe the following things -
> 
> 1. Reads are evenly distributed among 5 nodes.  CPUs remain under 5% utilized.
> 
> 2. We have 4gb block cache (30% block cache out of 12gb) setup. 3gb block 
> cache got filled up but around 1gb remained free. There are a large number of 
> cache eviction.
> 
> Questions to experts -
> 
> 1. If there are still 1gb of free block cache available, why is hbase 
> evicting the block from cache?
> 
> 4. We are seeing memory went up to 10gb three times before dropping sharply 
> to 5gb.
> 
> Any help is highly appreciable,
> 
> Thanks,
> Saurabh.
> 
> Confidentiality Notice:  The information contained in this message, including 
> any attachments hereto, may be confidential and is intended to be read only 
> by the individual or entity to whom this message is addressed. If the reader 
> of this message is not the intended recipient or an agent or designee of the 
> intended recipient, please note that any review, use, disclosure or 
> distribution of this message or its attachments, in any form, is strictly 
> prohibited.  If you have received this message in error, please immediately 
> notify the sender and/or notificati...@carrieriq.com and delete or destroy 
> any copy of this message and its attachments.

Re: RowLocks

[lars hofhansl]

Specifically the API has been removed because it had never actually worked 
correctly.


Rowlocks are used by RegionServers for intra-region operations.
As such they are ephemeral, in-memory constructs, that cannot reliably outlive 
a single RPC request.
The HTable rowlock API allowed you to create a rowlocks and hold it over 
multiple RPCs, which would break if f.e. a region is moved or split.

-- Lars

From: Ted Yu 
To: "user@hbase.apache.org"  
Sent: Wednesday, August 28, 2013 8:01 PM
Subject: Re: RowLocks


The API is no longer a public API

Thanks


On Wed, Aug 28, 2013 at 7:58 PM, Michael Segel wrote:

> Ted,
> Can you clarify...
> Do you mean the API is no longer a public API, or do you mean no more RLL
> for atomic writes?
>
> On Aug 28, 2013, at 5:18 PM, Ted Yu  wrote:
>
> > RowLock API has been removed in 0.96.
> >
> > Can you tell us your use case ?
> >
> >
> > On Wed, Aug 28, 2013 at 3:14 PM, Kristoffer Sjögren  >wrote:
> >
> >> Hi
> >>
> >> About the internals of locking a row in hbase.
> >>
> >> Does hbase row locks map one-to-one with a locks in zookeeper or are
> there
> >> any optimizations based on the fact that a row only exist on a single
> >> machine?
> >>
> >> Cheers,
> >> -Kristoffer
> >>
>
> The opinions expressed here are mine, while they may reflect a cognitive
> thought, that is purely accidental.
> Use at your own risk.
> Michael Segel
> michael_segel (AT) hotmail.com
>
>
>
>
>
>

Re: RowLocks

[Ted Yu]

The API is no longer a public API

Thanks


On Wed, Aug 28, 2013 at 7:58 PM, Michael Segel wrote:

> Ted,
> Can you clarify...
> Do you mean the API is no longer a public API, or do you mean no more RLL
> for atomic writes?
>
> On Aug 28, 2013, at 5:18 PM, Ted Yu  wrote:
>
> > RowLock API has been removed in 0.96.
> >
> > Can you tell us your use case ?
> >
> >
> > On Wed, Aug 28, 2013 at 3:14 PM, Kristoffer Sjögren  >wrote:
> >
> >> Hi
> >>
> >> About the internals of locking a row in hbase.
> >>
> >> Does hbase row locks map one-to-one with a locks in zookeeper or are
> there
> >> any optimizations based on the fact that a row only exist on a single
> >> machine?
> >>
> >> Cheers,
> >> -Kristoffer
> >>
>
> The opinions expressed here are mine, while they may reflect a cognitive
> thought, that is purely accidental.
> Use at your own risk.
> Michael Segel
> michael_segel (AT) hotmail.com
>
>
>
>
>
>

Re: RowLocks

[Michael Segel]

Ted, 
Can you clarify... 
Do you mean the API is no longer a public API, or do you mean no more RLL for 
atomic writes?

On Aug 28, 2013, at 5:18 PM, Ted Yu  wrote:

> RowLock API has been removed in 0.96.
> 
> Can you tell us your use case ?
> 
> 
> On Wed, Aug 28, 2013 at 3:14 PM, Kristoffer Sjögren wrote:
> 
>> Hi
>> 
>> About the internals of locking a row in hbase.
>> 
>> Does hbase row locks map one-to-one with a locks in zookeeper or are there
>> any optimizations based on the fact that a row only exist on a single
>> machine?
>> 
>> Cheers,
>> -Kristoffer
>> 

The opinions expressed here are mine, while they may reflect a cognitive 
thought, that is purely accidental. 
Use at your own risk. 
Michael Segel
michael_segel (AT) hotmail.com

Re: RowLocks

[Chris Perluss]

You could add an "isLocked" column to your row. When you want to lock or
update your row then use checkAndPut and check that "isLocked"=0.  When
unlocking your row then checkAndPut that "isLocked"=1.  You will have
effectively locked the row for the purposes of your application without
affecting HBase internal workings like performing a region split.

Chris
I want a distributed lock condition for doing certain operations that may
or may not be unrelated to hbase.


On Thu, Aug 29, 2013 at 12:18 AM, Ted Yu  wrote:

> RowLock API has been removed in 0.96.
>
> Can you tell us your use case ?
>
>
> On Wed, Aug 28, 2013 at 3:14 PM, Kristoffer Sjögren  >wrote:
>
> > Hi
> >
> > About the internals of locking a row in hbase.
> >
> > Does hbase row locks map one-to-one with a locks in zookeeper or are
> there
> > any optimizations based on the fact that a row only exist on a single
> > machine?
> >
> > Cheers,
> > -Kristoffer
> >
>

Re: Pack rows into a wide row for better performance?

[林煒清]

Oh , so they have them packed into one cell .  If so, now its reasonable
that they claim it speed up row seeking .
thanks a lot.


2013/8/28 Chris Perluss 

> Sorry, accidentally hit send. I'm guessing a 10 minute time slice would
> drop their space savings from 4-8x down to 2-4x.
> On Aug 27, 2013 11:30 PM, "Chris Perluss"  wrote:
>
> > I'm still kinda new to HBase so please excuse me if I am wrong.  I
> suspect
> > the reason has to do with a different slide from their presentation where
> > they run a job every hour to combine all the cells from the previous hour
> > into one cell.
> >
> > OpenTSDB has quite a long row key. It contains the metric name, the
> > timestamp, and numerous optional tags. If you wrote one metric every
> second
> > then you would write 3600 columns per row key. Since the row key is very
> > long, it uses quite a bit of space to store the same row key 3600 times.
> > By combining an hours worth of data into one cell OpenTMS claims they
> save
> > 4-8x of their storage.
> >
> > If they stayed with their original 10 minute time slice then they would
> > have to store their giant row key 6 times per hour instead of once. I'm
> > going to guess this
> > On Aug 27, 2013 10:50 PM, "林煒清"  wrote:
> >
> >> *Context*:
> >>
> >> Recently, I see openTSDB having their rows packed by period, thus end in
> >> ten to hundred columns per row. It claim that this design performs more
> >> efficient for row seeking.(on slide:Lessons learned from openTSDB)
> >>
> >> *My argument*:
> >>
> >>  If *a block of HFile *is indexed by the start key of itself, which the
> >> key
> >> is made of {row, cf, cq} , then I think read time for the specific Key
> >> should be the same for all tall-or-wide table case, since the physical
> >> storage is sorted by key, not only by rowkey.
> >>
> >>  So that under one column family the rowkey+column is a key as a whole,
> >> shift a part of the rowkey to the column is the same as shift a part of
> >> rowkey to the tail of the rowkey, vice versa.
> >>
> >> Follow this logic , under physical view the openTSDB did is just change
> >> key
> >> index by shifting a portion of timestamp bytes to position behind
> rowkey,
> >> that is column qualifier.
> >>
> >> *Question*:
> >>
> >> 1.When getting (get is a special scan, right?) a packed row worth of one
> >> hour, or scan over one hour range of rows, I don't see there could any
> >> performance improvement. So why openTSDB says packed row have better
> >> performance for row seeking?
> >>
> >> 2.Almost every doc & books all recommend tall table design and
> especially
> >> at book "HBase in Action", it says that ,among others, the consideration
> >> of
> >> reading performance is the reason why tall is adopting, though I still
> >> can't get it why?
> >>
> >> 3.Also that the KeyValues inside a block is searched by *linear* scan,
> and
> >> start key of blocks is by binary search , right?
> >>
> >> any hint is much appreciated.
> >>
> >
>

Re: Lease Exception Errors When Running Heavy Map Reduce Job

[Ameya Kanitkar]

Any ideas? Anyone?


On Wed, Aug 28, 2013 at 9:36 AM, Ameya Kanitkar  wrote:

> Thanks for your response.
>
> I checked namenode logs and I find following:
>
> 2013-08-28 15:25:24,025 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: recoverLease: recover
> lease [Lease.  Holder:
> DFSClient_hb_rs_smartdeals-hbase14-snc1.snc1,60020,1377700014053_-346895658_25,
> pendingcreates: 1],
> src=/hbase/.logs/smartdeals-hbase14-snc1.snc1,60020,1377700014053-splitting/smartdeals-hbase14-snc1.snc1%2C60020%2C1377700014053.1377700015413
> from client
> DFSClient_hb_rs_smartdeals-hbase14-snc1.snc1,60020,1377700014053_-346895658_25
> 2013-08-28 15:25:24,025 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Recovering
> lease=[Lease.  Holder:
> DFSClient_hb_rs_smartdeals-hbase14-snc1.snc1,60020,1377700014053_-346895658_25,
> pendingcreates: 1],
> src=/hbase/.logs/smartdeals-hbase14-snc1.snc1,60020,1377700014053-splitting/smartdeals-hbase14-snc1.snc1%2C60020%2C1377700014053.1377700015413
> 2013-08-28 15:25:24,025 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
> internalReleaseLease: All existing blocks are COMPLETE, lease removed, file
> closed.
>
> There are LeaseException errors on namenode as well:
> http://pastebin.com/4feVcL1F Not sure why its happening.
>
> I do not think I am ending up with any timeouts, as my jobs fail within
> couple of minutes, while all my time outs are 10 minutes+
> Not sure why above would
>
> Ameya
>
>
>
> On Wed, Aug 28, 2013 at 9:00 AM, Ted Yu  wrote:
>
>> From the log you posted on pastebin, I see the following.
>> Can you check namenode log to see what went wrong ?
>>
>>
>>1. Caused by:
>>org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease
>> on
>>
>>  
>> /hbase/.logs/smartdeals-hbase14-snc1.snc1,60020,1376944419197/smartdeals-hbase14-snc1.snc1%2C60020%2C1376944419197.1377699297514
>>File does not exist. [Lease.  Holder:
>>
>>  
>> DFSClient_hb_rs_smartdeals-hbase14-snc1.snc1,60020,1376944419197_-413917755_25,
>>pendingcreates: 1]
>>
>>
>>
>> On Wed, Aug 28, 2013 at 8:00 AM, Ameya Kanitkar 
>> wrote:
>>
>> > HI All,
>> >
>> > We have a very heavy map reduce job that goes over entire table with
>> over
>> > 1TB+ data in HBase and exports all data (Similar to Export job but with
>> > some additional custom code built in) to HDFS.
>> >
>> > However this job is not very stable, and often times we get following
>> error
>> > and job fails:
>> >
>> > org.apache.hadoop.hbase.regionserver.LeaseException:
>> > org.apache.hadoop.hbase.regionserver.LeaseException: lease
>> > '-4456594242606811626' does not exist
>> > at
>> > org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2429)
>> > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1400)
>> >
>> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> > Method)
>> > at
>> >
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>> > at java.lang.reflect.Constructor.newInstance(Constructor.
>> >
>> >
>> > Here are more detailed logs on the RS: http://pastebin.com/xaHF4ksb
>> >
>> > We have changed following settings in HBase to counter this problem
>> > but issue persists:
>> >
>> > 
>> > 
>> > hbase.regionserver.lease.period
>> > 90
>> > 
>> >
>> > 
>> > 
>> > hbase.rpc.timeout
>> > 90
>> > 
>> >
>> >
>> > We also reduced number of mappers per RS less than available CPU's on
>> the
>> > box.
>> >
>> > We also observed that problem once happens, happens multiple times on
>> > the same RS. All other regions are unaffected. But different RS
>> > observes this problem on different days. There is no particular region
>> > causing this either.
>> >
>> > We are running: 0.94.2 with cdh4.2.0
>> >
>> > Any ideas?
>> >
>> >
>> > Ameya
>> >
>>
>
>

RE: experiencing high latency for few reads in HBase

[Vladimir Rodionov]

Increasing Java heap size will make latency worse, actually.
You can't guarantee 1 sec max latency if run Java app (unless your heap size is 
much less than 1GB). 
I have never heard about strict maximum latency limit. Usually , its 99% , 99.9 
or 99.99% query percentiles.

You can greatly reduce your 99.xxx% percentile latency by storing you data in 2 
replicas to two different region servers.
Issue two read operations to those two region servers in parallel and get the 
first response. Probability theory states that  probability 
of two independent events (slow requests) is  the product of event's 
probabilities themselves. 


Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Saurabh Yahoo [saurabh...@yahoo.com]
Sent: Wednesday, August 28, 2013 4:18 PM
To: user@hbase.apache.org
Subject: Re: experiencing high latency for few reads in HBase

Thanks Kiru,

Scan is not an option for our use cases.  Our read is pretty random.

Any other suggestion to bring down the latency.

Thanks,
Saurabh.


On Aug 28, 2013, at 7:01 PM, Kiru Pakkirisamy  wrote:

> Saurabh, we are able to 600K rowxcolumns in 400 msec. We have put what was a 
> 40million row table as 400K rows and columns. We Get about 100 of the rows 
> from this 400K , do quite a bit of calculations in the coprocessor (almost a 
> group-order by) and return in this time.
> Maybe should consider replacing the MultiGets with Scan with Filter. I like 
> the FuzzyRowFilter even though you might need to match with exact key. It 
> works only with fixed length key.
> (I do have an issue right now, it is not scaling to multiple clients.)
>
> Regards,
> - kiru
>
>
> Kiru Pakkirisamy | webcloudtech.wordpress.com
>
>
> 
> From: Saurabh Yahoo 
> To: "user@hbase.apache.org" 
> Cc: "user@hbase.apache.org" 
> Sent: Wednesday, August 28, 2013 3:20 PM
> Subject: Re: experiencing high latency for few reads in HBase
>
>
> Thanks Kitu. We need less than 1 sec latency.
>
> We are using both muliGet and get.
>
> We have three concurrent clients running 10 threads each. ( that makes total 
> 30 concurrent clients).
>
> Thanks,
> Saurabh.
>
> On Aug 28, 2013, at 4:30 PM, Kiru Pakkirisamy  
> wrote:
>
>> Right 4 sec is good.
>> @Saurabh - so your read is - getting 20 out of 25 millions rows ?. Is this a 
>> Get or a Scan ?
>> BTW, in this stress test how many concurrent clients do you have ?
>>
>> Regards,
>> - kiru
>>
>>
>> 
>> From: Vladimir Rodionov 
>> To: "user@hbase.apache.org" 
>> Sent: Wednesday, August 28, 2013 12:15 PM
>> Subject: RE: experiencing high latency for few reads in HBase
>>
>>
>> 1. 4 sec max latency is not that bad taking into account 12GB heap.  It can 
>> be much larger. What is your SLA?
>> 2. Block evictions is the result of a poor cache hit rate and the root cause 
>> of a periodic stop-the-world GC pauses (max latencies
>>  latencies you have been observing in the test)
>> 3. Block cache consists of 3 parts (25% young generation, 50% - tenured, 25% 
>> - permanent). Permanent part is for CF with
>> IN_MEMORY = true (you can specify this when you create CF).  Block first 
>> stored in 'young gen' space, then gets promoted to 'tenured gen' space
>> (or gets evicted). May be your 'perm gen' space is underutilized? This is 
>> exact 25% of 4GB (1GB). Although HBase LruBlockCache should use all the 
>> space allocated for block cache -
>> there is no guarantee (as usual). If you don have in_memory column families 
>> you may decrease
>>
>>
>>
>> Best regards,
>> Vladimir Rodionov
>> Principal Platform Engineer
>> Carrier IQ, www.carrieriq.com
>> e-mail: vrodio...@carrieriq.com
>>
>> 
>> From: Saurabh Yahoo [saurabh...@yahoo.com]
>> Sent: Wednesday, August 28, 2013 5:10 AM
>> To: user@hbase.apache.org
>> Subject: experiencing high latency for few reads in HBase
>>
>> Hi,
>>
>> We are running a stress test in our 5 node cluster and we are getting the 
>> expected mean latency of 10ms. But we are seeing around 20 reads out of 25 
>> million reads having latency more than 4 seconds. Can anyone provide the 
>> insight what we can do to meet below second SLA for each and every read?
>>
>> We observe the following things -
>>
>> 1. Reads are evenly distributed among 5 nodes.  CPUs remain under 5% 
>> utilized.
>>
>> 2. We have 4gb block cache (30% block cache out of 12gb) setup. 3gb block 
>> cache got filled up but around 1gb remained free. There are a large number 
>> of cache eviction.
>>
>> Questions to experts -
>>
>> 1. If there are still 1gb of free block cache available, why is hbase 
>> evicting the block from cache?
>>
>> 4. We are seeing memory went up to 10gb three times before dropping sharply 
>> to 5gb.
>>
>> Any help is highly appreciable,
>>
>> Thanks,
>> Saurabh.
>>
>> Confidentiality Notice:  The information contained in 

Re: experiencing high latency for few reads in HBase

[Saurabh Yahoo]

Thanks Kiru,

Scan is not an option for our use cases.  Our read is pretty random.

Any other suggestion to bring down the latency.   

Thanks,
Saurabh. 


On Aug 28, 2013, at 7:01 PM, Kiru Pakkirisamy  wrote:

> Saurabh, we are able to 600K rowxcolumns in 400 msec. We have put what was a 
> 40million row table as 400K rows and columns. We Get about 100 of the rows 
> from this 400K , do quite a bit of calculations in the coprocessor (almost a 
> group-order by) and return in this time.
> Maybe should consider replacing the MultiGets with Scan with Filter. I like 
> the FuzzyRowFilter even though you might need to match with exact key. It 
> works only with fixed length key.
> (I do have an issue right now, it is not scaling to multiple clients.)
>  
> Regards,
> - kiru
> 
> 
> Kiru Pakkirisamy | webcloudtech.wordpress.com
> 
> 
> 
> From: Saurabh Yahoo 
> To: "user@hbase.apache.org"  
> Cc: "user@hbase.apache.org"  
> Sent: Wednesday, August 28, 2013 3:20 PM
> Subject: Re: experiencing high latency for few reads in HBase 
> 
> 
> Thanks Kitu. We need less than 1 sec latency.  
> 
> We are using both muliGet and get. 
> 
> We have three concurrent clients running 10 threads each. ( that makes total 
> 30 concurrent clients).
> 
> Thanks,
> Saurabh.  
> 
> On Aug 28, 2013, at 4:30 PM, Kiru Pakkirisamy  
> wrote:
> 
>> Right 4 sec is good.  
>> @Saurabh - so your read is - getting 20 out of 25 millions rows ?. Is this a 
>> Get or a Scan ?
>> BTW, in this stress test how many concurrent clients do you have ? 
>>   
>> Regards,
>> - kiru
>> 
>> 
>> 
>> From: Vladimir Rodionov 
>> To: "user@hbase.apache.org"  
>> Sent: Wednesday, August 28, 2013 12:15 PM
>> Subject: RE: experiencing high latency for few reads in HBase 
>> 
>> 
>> 1. 4 sec max latency is not that bad taking into account 12GB heap.  It can 
>> be much larger. What is your SLA?
>> 2. Block evictions is the result of a poor cache hit rate and the root cause 
>> of a periodic stop-the-world GC pauses (max latencies
>>  latencies you have been observing in the test)
>> 3. Block cache consists of 3 parts (25% young generation, 50% - tenured, 25% 
>> - permanent). Permanent part is for CF with
>> IN_MEMORY = true (you can specify this when you create CF).  Block first 
>> stored in 'young gen' space, then gets promoted to 'tenured gen' space
>> (or gets evicted). May be your 'perm gen' space is underutilized? This is 
>> exact 25% of 4GB (1GB). Although HBase LruBlockCache should use all the 
>> space allocated for block cache -
>> there is no guarantee (as usual). If you don have in_memory column families 
>> you may decrease
>> 
>> 
>> 
>> Best regards,
>> Vladimir Rodionov
>> Principal Platform Engineer
>> Carrier IQ, www.carrieriq.com
>> e-mail: vrodio...@carrieriq.com
>> 
>> 
>> From: Saurabh Yahoo [saurabh...@yahoo.com]
>> Sent: Wednesday, August 28, 2013 5:10 AM
>> To: user@hbase.apache.org
>> Subject: experiencing high latency for few reads in HBase
>> 
>> Hi,
>> 
>> We are running a stress test in our 5 node cluster and we are getting the 
>> expected mean latency of 10ms. But we are seeing around 20 reads out of 25 
>> million reads having latency more than 4 seconds. Can anyone provide the 
>> insight what we can do to meet below second SLA for each and every read?
>> 
>> We observe the following things -
>> 
>> 1. Reads are evenly distributed among 5 nodes.  CPUs remain under 5% 
>> utilized.
>> 
>> 2. We have 4gb block cache (30% block cache out of 12gb) setup. 3gb block 
>> cache got filled up but around 1gb remained free. There are a large number 
>> of cache eviction.
>> 
>> Questions to experts -
>> 
>> 1. If there are still 1gb of free block cache available, why is hbase 
>> evicting the block from cache?
>> 
>> 4. We are seeing memory went up to 10gb three times before dropping sharply 
>> to 5gb.
>> 
>> Any help is highly appreciable,
>> 
>> Thanks,
>> Saurabh.
>> 
>> Confidentiality Notice:  The information contained in this message, 
>> including any attachments hereto, may be confidential and is intended to be 
>> read only by the individual or entity to whom this message is addressed. If 
>> the reader of this message is not the intended recipient or an agent or 
>> designee of the intended recipient, please note that any review, use, 
>> disclosure or distribution of this message or its attachments, in any form, 
>> is strictly prohibited.  If you have received this message in error, please 
>> immediately notify the sender and/or notificati...@carrieriq.com and delete 
>> or destroy any copy of this message and its attachments.

Re: KeyValue.parseColumn

[Vandana Ayyalasomayajula]

Thanks Stack for following up.

On Aug 28, 2013, at 3:31 PM, Stack wrote:

> On Wed, Aug 28, 2013 at 12:46 PM, Vandana Ayyalasomayajula <
> avand...@yahoo-inc.com> wrote:
> 
>> Hi All,
>> 
>> I have been looking at the "parseColumn" method in  the KeyValue class of
>> HBase. The javadoc of that method
>> does not recommend that method be used. I wanted to know if there is any
>> other existing API which can be used in
>> place of the above method.
>> 
> 
> Yeah, we want to ween folks of specifying columns as "family:qualifier" and
> instead have them specify the two dimensions individually.  Go ahead and
> use the method if you cannot do this.
> 
> St.Ack

Re: experiencing high latency for few reads in HBase

[Kiru Pakkirisamy]

Saurabh, we are able to 600K rowxcolumns in 400 msec. We have put what was a 
40million row table as 400K rows and columns. We Get about 100 of the rows from 
this 400K , do quite a bit of calculations in the coprocessor (almost a 
group-order by) and return in this time.
Maybe should consider replacing the MultiGets with Scan with Filter. I like the 
FuzzyRowFilter even though you might need to match with exact key. It works 
only with fixed length key.
(I do have an issue right now, it is not scaling to multiple clients.)
 
Regards,
- kiru


Kiru Pakkirisamy | webcloudtech.wordpress.com



 From: Saurabh Yahoo 
To: "user@hbase.apache.org"  
Cc: "user@hbase.apache.org"  
Sent: Wednesday, August 28, 2013 3:20 PM
Subject: Re: experiencing high latency for few reads in HBase 
 

Thanks Kitu. We need less than 1 sec latency.  

We are using both muliGet and get. 

We have three concurrent clients running 10 threads each. ( that makes total 30 
concurrent clients).

Thanks,
Saurabh.  

On Aug 28, 2013, at 4:30 PM, Kiru Pakkirisamy  wrote:

> Right 4 sec is good.  
> @Saurabh - so your read is - getting 20 out of 25 millions rows ?. Is this a 
> Get or a Scan ?
> BTW, in this stress test how many concurrent clients do you have ? 
>  
> Regards,
> - kiru
> 
> 
> 
> From: Vladimir Rodionov 
> To: "user@hbase.apache.org"  
> Sent: Wednesday, August 28, 2013 12:15 PM
> Subject: RE: experiencing high latency for few reads in HBase 
> 
> 
> 1. 4 sec max latency is not that bad taking into account 12GB heap.  It can 
> be much larger. What is your SLA?
> 2. Block evictions is the result of a poor cache hit rate and the root cause 
> of a periodic stop-the-world GC pauses (max latencies
>     latencies you have been observing in the test)
> 3. Block cache consists of 3 parts (25% young generation, 50% - tenured, 25% 
> - permanent). Permanent part is for CF with
> IN_MEMORY = true (you can specify this when you create CF).  Block first 
> stored in 'young gen' space, then gets promoted to 'tenured gen' space
> (or gets evicted). May be your 'perm gen' space is underutilized? This is 
> exact 25% of 4GB (1GB). Although HBase LruBlockCache should use all the space 
> allocated for block cache -
> there is no guarantee (as usual). If you don have in_memory column families 
> you may decrease
> 
> 
> 
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodio...@carrieriq.com
> 
> 
> From: Saurabh Yahoo [saurabh...@yahoo.com]
> Sent: Wednesday, August 28, 2013 5:10 AM
> To: user@hbase.apache.org
> Subject: experiencing high latency for few reads in HBase
> 
> Hi,
> 
> We are running a stress test in our 5 node cluster and we are getting the 
> expected mean latency of 10ms. But we are seeing around 20 reads out of 25 
> million reads having latency more than 4 seconds. Can anyone provide the 
> insight what we can do to meet below second SLA for each and every read?
> 
> We observe the following things -
> 
> 1. Reads are evenly distributed among 5 nodes.  CPUs remain under 5% utilized.
> 
> 2. We have 4gb block cache (30% block cache out of 12gb) setup. 3gb block 
> cache got filled up but around 1gb remained free. There are a large number of 
> cache eviction.
> 
> Questions to experts -
> 
> 1. If there are still 1gb of free block cache available, why is hbase 
> evicting the block from cache?
> 
> 4. We are seeing memory went up to 10gb three times before dropping sharply 
> to 5gb.
> 
> Any help is highly appreciable,
> 
> Thanks,
> Saurabh.
> 
> Confidentiality Notice:  The information contained in this message, including 
> any attachments hereto, may be confidential and is intended to be read only 
> by the individual or entity to whom this message is addressed. If the reader 
> of this message is not the intended recipient or an agent or designee of the 
> intended recipient, please note that any review, use, disclosure or 
> distribution of this message or its attachments, in any form, is strictly 
> prohibited.  If you have received this message in error, please immediately 
> notify the sender and/or notificati...@carrieriq.com and delete or destroy 
> any copy of this message and its attachments.

Re: KeyValue.parseColumn

[Stack]

On Wed, Aug 28, 2013 at 12:46 PM, Vandana Ayyalasomayajula <
avand...@yahoo-inc.com> wrote:

> Hi All,
>
> I have been looking at the "parseColumn" method in  the KeyValue class of
> HBase. The javadoc of that method
> does not recommend that method be used. I wanted to know if there is any
> other existing API which can be used in
> place of the above method.
>

Yeah, we want to ween folks of specifying columns as "family:qualifier" and
instead have them specify the two dimensions individually.  Go ahead and
use the method if you cannot do this.

St.Ack

Re: RowLocks

[Kristoffer Sjögren]

I want a distributed lock condition for doing certain operations that may
or may not be unrelated to hbase.


On Thu, Aug 29, 2013 at 12:18 AM, Ted Yu  wrote:

> RowLock API has been removed in 0.96.
>
> Can you tell us your use case ?
>
>
> On Wed, Aug 28, 2013 at 3:14 PM, Kristoffer Sjögren  >wrote:
>
> > Hi
> >
> > About the internals of locking a row in hbase.
> >
> > Does hbase row locks map one-to-one with a locks in zookeeper or are
> there
> > any optimizations based on the fact that a row only exist on a single
> > machine?
> >
> > Cheers,
> > -Kristoffer
> >
>

Re: RowLocks

[Jean-Marc Spaggiari]

Worst case you can use ZK to do the same if you only need that from time to
time?
Le 2013-08-28 18:19, "Ted Yu"  a écrit :

> RowLock API has been removed in 0.96.
>
> Can you tell us your use case ?
>
>
> On Wed, Aug 28, 2013 at 3:14 PM, Kristoffer Sjögren  >wrote:
>
> > Hi
> >
> > About the internals of locking a row in hbase.
> >
> > Does hbase row locks map one-to-one with a locks in zookeeper or are
> there
> > any optimizations based on the fact that a row only exist on a single
> > machine?
> >
> > Cheers,
> > -Kristoffer
> >
>

Re: experiencing high latency for few reads in HBase

[Saurabh Yahoo]

Thanks Kitu. We need less than 1 sec latency.  

We are using both muliGet and get. 

We have three concurrent clients running 10 threads each. ( that makes total 30 
concurrent clients).

Thanks,
Saurabh.  

On Aug 28, 2013, at 4:30 PM, Kiru Pakkirisamy  wrote:

> Right 4 sec is good.  
> @Saurabh - so your read is - getting 20 out of 25 millions rows ?. Is this a 
> Get or a Scan ?
> BTW, in this stress test how many concurrent clients do you have ? 
>  
> Regards,
> - kiru
> 
> 
> 
> From: Vladimir Rodionov 
> To: "user@hbase.apache.org"  
> Sent: Wednesday, August 28, 2013 12:15 PM
> Subject: RE: experiencing high latency for few reads in HBase 
> 
> 
> 1. 4 sec max latency is not that bad taking into account 12GB heap.  It can 
> be much larger. What is your SLA?
> 2. Block evictions is the result of a poor cache hit rate and the root cause 
> of a periodic stop-the-world GC pauses (max latencies
> latencies you have been observing in the test)
> 3. Block cache consists of 3 parts (25% young generation, 50% - tenured, 25% 
> - permanent). Permanent part is for CF with
> IN_MEMORY = true (you can specify this when you create CF).  Block first 
> stored in 'young gen' space, then gets promoted to 'tenured gen' space
> (or gets evicted). May be your 'perm gen' space is underutilized? This is 
> exact 25% of 4GB (1GB). Although HBase LruBlockCache should use all the space 
> allocated for block cache -
> there is no guarantee (as usual). If you don have in_memory column families 
> you may decrease
> 
> 
> 
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodio...@carrieriq.com
> 
> 
> From: Saurabh Yahoo [saurabh...@yahoo.com]
> Sent: Wednesday, August 28, 2013 5:10 AM
> To: user@hbase.apache.org
> Subject: experiencing high latency for few reads in HBase
> 
> Hi,
> 
> We are running a stress test in our 5 node cluster and we are getting the 
> expected mean latency of 10ms. But we are seeing around 20 reads out of 25 
> million reads having latency more than 4 seconds. Can anyone provide the 
> insight what we can do to meet below second SLA for each and every read?
> 
> We observe the following things -
> 
> 1. Reads are evenly distributed among 5 nodes.  CPUs remain under 5% utilized.
> 
> 2. We have 4gb block cache (30% block cache out of 12gb) setup. 3gb block 
> cache got filled up but around 1gb remained free. There are a large number of 
> cache eviction.
> 
> Questions to experts -
> 
> 1. If there are still 1gb of free block cache available, why is hbase 
> evicting the block from cache?
> 
> 4. We are seeing memory went up to 10gb three times before dropping sharply 
> to 5gb.
> 
> Any help is highly appreciable,
> 
> Thanks,
> Saurabh.
> 
> Confidentiality Notice:  The information contained in this message, including 
> any attachments hereto, may be confidential and is intended to be read only 
> by the individual or entity to whom this message is addressed. If the reader 
> of this message is not the intended recipient or an agent or designee of the 
> intended recipient, please note that any review, use, disclosure or 
> distribution of this message or its attachments, in any form, is strictly 
> prohibited.  If you have received this message in error, please immediately 
> notify the sender and/or notificati...@carrieriq.com and delete or destroy 
> any copy of this message and its attachments.

Re: RowLocks

[Ted Yu]

RowLock API has been removed in 0.96.

Can you tell us your use case ?


On Wed, Aug 28, 2013 at 3:14 PM, Kristoffer Sjögren wrote:

> Hi
>
> About the internals of locking a row in hbase.
>
> Does hbase row locks map one-to-one with a locks in zookeeper or are there
> any optimizations based on the fact that a row only exist on a single
> machine?
>
> Cheers,
> -Kristoffer
>

Re: experiencing high latency for few reads in HBase

[Saurabh Yahoo]

Hi Vlad,

Thanks for your response.

1. Our SLA is less than one sec. we cannot afford latency more than 1 sec. 

We can increase heap size if that help, we have enough memory on server. What 
would be the optimal heap size? 

2. Cache hit ratio is 95%.  One thing I don't understand that we have allocated 
only 4gb for block cache out of 12gb. That left 8gb for rest of JVM. There is 
no write. Memcache is empty. Is 8gb not enough for hbase to process the 
requests? What are the most memory consuming objects in region server? 

3. We will change the cf to IN_memory and report back performance difference.  

Thanks,
Saurabh. 

On Aug 28, 2013, at 3:15 PM, Vladimir Rodionov  wrote:

> 1. 4 sec max latency is not that bad taking into account 12GB heap.  It can 
> be much larger. What is your SLA?
> 2. Block evictions is the result of a poor cache hit rate and the root cause 
> of a periodic stop-the-world GC pauses (max latencies
>latencies you have been observing in the test)
> 3. Block cache consists of 3 parts (25% young generation, 50% - tenured, 25% 
> - permanent). Permanent part is for CF with
> IN_MEMORY = true (you can specify this when you create CF).  Block first 
> stored in 'young gen' space, then gets promoted to 'tenured gen' space
> (or gets evicted). May be your 'perm gen' space is underutilized? This is 
> exact 25% of 4GB (1GB). Although HBase LruBlockCache should use all the space 
> allocated for block cache -
> there is no guarantee (as usual). If you don have in_memory column families 
> you may decrease
> 
> 
> 
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodio...@carrieriq.com
> 
> 
> From: Saurabh Yahoo [saurabh...@yahoo.com]
> Sent: Wednesday, August 28, 2013 5:10 AM
> To: user@hbase.apache.org
> Subject: experiencing high latency for few reads in HBase
> 
> Hi,
> 
> We are running a stress test in our 5 node cluster and we are getting the 
> expected mean latency of 10ms. But we are seeing around 20 reads out of 25 
> million reads having latency more than 4 seconds. Can anyone provide the 
> insight what we can do to meet below second SLA for each and every read?
> 
> We observe the following things -
> 
> 1. Reads are evenly distributed among 5 nodes.  CPUs remain under 5% utilized.
> 
> 2. We have 4gb block cache (30% block cache out of 12gb) setup. 3gb block 
> cache got filled up but around 1gb remained free. There are a large number of 
> cache eviction.
> 
> Questions to experts -
> 
> 1. If there are still 1gb of free block cache available, why is hbase 
> evicting the block from cache?
> 
> 4. We are seeing memory went up to 10gb three times before dropping sharply 
> to 5gb.
> 
> Any help is highly appreciable,
> 
> Thanks,
> Saurabh.
> 
> Confidentiality Notice:  The information contained in this message, including 
> any attachments hereto, may be confidential and is intended to be read only 
> by the individual or entity to whom this message is addressed. If the reader 
> of this message is not the intended recipient or an agent or designee of the 
> intended recipient, please note that any review, use, disclosure or 
> distribution of this message or its attachments, in any form, is strictly 
> prohibited.  If you have received this message in error, please immediately 
> notify the sender and/or notificati...@carrieriq.com and delete or destroy 
> any copy of this message and its attachments.

RowLocks

[Kristoffer Sjögren]

Hi

About the internals of locking a row in hbase.

Does hbase row locks map one-to-one with a locks in zookeeper or are there
any optimizations based on the fact that a row only exist on a single
machine?

Cheers,
-Kristoffer

Re: How to enable metrics2?

[Elliott Clark]

On Wed, Aug 28, 2013 at 6:55 AM, Ionut Ignatescu
 wrote:
> MetricsSink that export metrics in a custom format. This solution runs

Metrics2 support in HBase will be released in 0.96.  0.94.x versions
of HBase still use the older metrics system.

Re: Coprocessor responseTooSlow error messages

[Kiru Pakkirisamy]

Ted,
There are no error message. Only a WARN. It lists all the arguments to the 
call. 
It seems like a resource configuration issue. I am unable to get past 60 
concurrent clients or so.
And I did set the rpc handler count to 400.
(I just deleted the log file so I have to redo it again, it takes a while to 
reload my db as well)
 
Regards,
- kiru


Kiru Pakkirisamy | webcloudtech.wordpress.com



 From: Ted Yu 
To: "user@hbase.apache.org" ; Kiru Pakkirisamy 
 
Sent: Wednesday, August 28, 2013 12:47 PM
Subject: Re: Coprocessor responseTooSlow error messages
 

Can you post error message ?

Thanks


On Wed, Aug 28, 2013 at 12:00 PM, Kiru Pakkirisamy <
kirupakkiris...@yahoo.com> wrote:

> I keep getting these error message when I run multiple clients. For a
> single client, the same table/query gets done in 400 msec. But for 60
> clients it jumps to 10 secs (1msec). Any ideas on where the bottle neck
> could be ? Or how to go about debugging this.
>
> Regards,
> - kiru
>
>
> Kiru Pakkirisamy | webcloudtech.wordpress.com

Re: experiencing high latency for few reads in HBase

[Kiru Pakkirisamy]

Right 4 sec is good.  
@Saurabh - so your read is - getting 20 out of 25 millions rows ?. Is this a 
Get or a Scan ?
BTW, in this stress test how many concurrent clients do you have ? 
 
Regards,
- kiru



 From: Vladimir Rodionov 
To: "user@hbase.apache.org"  
Sent: Wednesday, August 28, 2013 12:15 PM
Subject: RE: experiencing high latency for few reads in HBase 
 

1. 4 sec max latency is not that bad taking into account 12GB heap.  It can be 
much larger. What is your SLA?
2. Block evictions is the result of a poor cache hit rate and the root cause of 
a periodic stop-the-world GC pauses (max latencies
    latencies you have been observing in the test)
3. Block cache consists of 3 parts (25% young generation, 50% - tenured, 25% - 
permanent). Permanent part is for CF with
IN_MEMORY = true (you can specify this when you create CF).  Block first stored 
in 'young gen' space, then gets promoted to 'tenured gen' space
(or gets evicted). May be your 'perm gen' space is underutilized? This is exact 
25% of 4GB (1GB). Although HBase LruBlockCache should use all the space 
allocated for block cache -
there is no guarantee (as usual). If you don have in_memory column families you 
may decrease



Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Saurabh Yahoo [saurabh...@yahoo.com]
Sent: Wednesday, August 28, 2013 5:10 AM
To: user@hbase.apache.org
Subject: experiencing high latency for few reads in HBase

Hi,

We are running a stress test in our 5 node cluster and we are getting the 
expected mean latency of 10ms. But we are seeing around 20 reads out of 25 
million reads having latency more than 4 seconds. Can anyone provide the 
insight what we can do to meet below second SLA for each and every read?

We observe the following things -

1. Reads are evenly distributed among 5 nodes.  CPUs remain under 5% utilized.

2. We have 4gb block cache (30% block cache out of 12gb) setup. 3gb block cache 
got filled up but around 1gb remained free. There are a large number of cache 
eviction.

Questions to experts -

1. If there are still 1gb of free block cache available, why is hbase evicting 
the block from cache?

4. We are seeing memory went up to 10gb three times before dropping sharply to 
5gb.

Any help is highly appreciable,

Thanks,
Saurabh.

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

KeyValue.parseColumn

[Vandana Ayyalasomayajula]

Hi All, 

I have been looking at the "parseColumn" method in  the KeyValue class of 
HBase. The javadoc of that method
does not recommend that method be used. I wanted to know if there is any other 
existing API which can be used in 
place of the above method. 


Thanks
Vandana

Re: Coprocessor responseTooSlow error messages

[Ted Yu]

Can you post error message ?

Thanks


On Wed, Aug 28, 2013 at 12:00 PM, Kiru Pakkirisamy <
kirupakkiris...@yahoo.com> wrote:

> I keep getting these error message when I run multiple clients. For a
> single client, the same table/query gets done in 400 msec. But for 60
> clients it jumps to 10 secs (1msec). Any ideas on where the bottle neck
> could be ? Or how to go about debugging this.
>
> Regards,
> - kiru
>
>
> Kiru Pakkirisamy | webcloudtech.wordpress.com

RE: experiencing high latency for few reads in HBase

[Vladimir Rodionov]

Just ignore last part: 'If you don have in_memory column families you may 
decrease'

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Vladimir Rodionov
Sent: Wednesday, August 28, 2013 12:15 PM
To: user@hbase.apache.org
Subject: RE: experiencing high latency for few reads in HBase

1. 4 sec max latency is not that bad taking into account 12GB heap.  It can be 
much larger. What is your SLA?
2. Block evictions is the result of a poor cache hit rate and the root cause of 
a periodic stop-the-world GC pauses (max latencies
latencies you have been observing in the test)
3. Block cache consists of 3 parts (25% young generation, 50% - tenured, 25% - 
permanent). Permanent part is for CF with
IN_MEMORY = true (you can specify this when you create CF).  Block first stored 
in 'young gen' space, then gets promoted to 'tenured gen' space
(or gets evicted). May be your 'perm gen' space is underutilized? This is exact 
25% of 4GB (1GB). Although HBase LruBlockCache should use all the space 
allocated for block cache -
there is no guarantee (as usual). If you don have in_memory column families you 
may decrease



Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Saurabh Yahoo [saurabh...@yahoo.com]
Sent: Wednesday, August 28, 2013 5:10 AM
To: user@hbase.apache.org
Subject: experiencing high latency for few reads in HBase

Hi,

We are running a stress test in our 5 node cluster and we are getting the 
expected mean latency of 10ms. But we are seeing around 20 reads out of 25 
million reads having latency more than 4 seconds. Can anyone provide the 
insight what we can do to meet below second SLA for each and every read?

We observe the following things -

1. Reads are evenly distributed among 5 nodes.  CPUs remain under 5% utilized.

2. We have 4gb block cache (30% block cache out of 12gb) setup. 3gb block cache 
got filled up but around 1gb remained free. There are a large number of cache 
eviction.

Questions to experts -

1. If there are still 1gb of free block cache available, why is hbase evicting 
the block from cache?

4. We are seeing memory went up to 10gb three times before dropping sharply to 
5gb.

Any help is highly appreciable,

Thanks,
Saurabh.

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: experiencing high latency for few reads in HBase

[Vladimir Rodionov]

1. 4 sec max latency is not that bad taking into account 12GB heap.  It can be 
much larger. What is your SLA?
2. Block evictions is the result of a poor cache hit rate and the root cause of 
a periodic stop-the-world GC pauses (max latencies
latencies you have been observing in the test)
3. Block cache consists of 3 parts (25% young generation, 50% - tenured, 25% - 
permanent). Permanent part is for CF with
IN_MEMORY = true (you can specify this when you create CF).  Block first stored 
in 'young gen' space, then gets promoted to 'tenured gen' space
(or gets evicted). May be your 'perm gen' space is underutilized? This is exact 
25% of 4GB (1GB). Although HBase LruBlockCache should use all the space 
allocated for block cache -
there is no guarantee (as usual). If you don have in_memory column families you 
may decrease



Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Saurabh Yahoo [saurabh...@yahoo.com]
Sent: Wednesday, August 28, 2013 5:10 AM
To: user@hbase.apache.org
Subject: experiencing high latency for few reads in HBase

Hi,

We are running a stress test in our 5 node cluster and we are getting the 
expected mean latency of 10ms. But we are seeing around 20 reads out of 25 
million reads having latency more than 4 seconds. Can anyone provide the 
insight what we can do to meet below second SLA for each and every read?

We observe the following things -

1. Reads are evenly distributed among 5 nodes.  CPUs remain under 5% utilized.

2. We have 4gb block cache (30% block cache out of 12gb) setup. 3gb block cache 
got filled up but around 1gb remained free. There are a large number of cache 
eviction.

Questions to experts -

1. If there are still 1gb of free block cache available, why is hbase evicting 
the block from cache?

4. We are seeing memory went up to 10gb three times before dropping sharply to 
5gb.

Any help is highly appreciable,

Thanks,
Saurabh.

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

Coprocessor responseTooSlow error messages

[Kiru Pakkirisamy]

I keep getting these error message when I run multiple clients. For a single 
client, the same table/query gets done in 400 msec. But for 60 clients it jumps 
to 10 secs (1msec). Any ideas on where the bottle neck could be ? Or how to 
go about debugging this.
 
Regards,
- kiru


Kiru Pakkirisamy | webcloudtech.wordpress.com

HBase indexing and updating

[Flavio Pompermaier]

Hi to everybody,
I have two questions:

- My HBase table is composed by a UUID as a key and xml as content in a
single column.
  Which is at the moment the best option to read all those xml, deserialize
to their object representation and add them to Solr (or another indexing
system)?
 The problem is that the maps could flood an indexing syustem if no queuing
system is designed in front of it,. My solution was to design a RabbitMQ
solution to make it scale (I'm able to index 8 millions of objects in about
30 minutes on a Solr 3.6.1) using 24 consumers on a single queue. Is there
a better solution to index my column content?

- If I have to modify all objects in my table, is it possible to make a put
on the row that the map function is reading? How is the best way to make
massive read-and-update?

Best,
Flavio

Re: Newbie in hbase Trying to run an example

[Doug Meil]

"cf" in this example is a column family, and this needs to exist in the
tables (both input and output) before the job is submitted.





On 8/26/13 3:01 PM, "jamal sasha"  wrote:

>Hi,
>  I am new to hbase, so few noob questions.
>
>So, I created a table in hbase:
>A quick scan gives me the following:
>hbase(main):001:0> scan 'test'
>ROW  COLUMN+CELL
>
>
> row1column=cf:word,
>timestamp=1377298314160, value=foo
>
> row2column=cf:word,
>timestamp=1377298326124, value=bar
>
> row3column=cf:word,
>timestamp=1377298332856, value=bar foo
>
> row4column=cf:word,
>timestamp=1377298347602, value=bar world foo
>
>Now, I want to do the word count and write the result back to another
>table
>in hbase
>So I followed the code given below:
>http://hbase.apache.org/book.html#mapreduce
>Snapshot in the end:
>Now, I am getting an error
>
>java.lang.NullPointerException
>at java.lang.String.(String.java:601)
>at org.rdf.HBaseExperiment$MyMapper.map(HBaseExperiment.java:42)
>at org.rdf.HBaseExperiment$MyMapper.map(HBaseExperiment.java:1)
>at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>at java.security.AccessController.doPrivileged(Native Method)
>at javax.security.auth.Subject.doAs(Subject.java:416)
>at
>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
>java:1093)
>at org.apache.hadoop.mapred.Child.main(Child.java:249)
>
>Line 42 points to
>*public static final byte[] ATTR1 = "attr1".getBytes();*
>
>Now I think attr1 is family qualifier.
>I am wondering, what exactly is a family qualifier?
>Do I need to set something while creating a table just like I did "cf"
>when
>I was creating the table.
>Similiarly what do I need to do on the "output" table as well?
>So, what I am saying is.. what do I need to to on hbase shell so that I
>can
>run this word count example?
>Thanks
>
>
>
>
>
>import java.io.IOException;
>import java.util.Date;
>
>import org.apache.hadoop.conf.Configuration;
>import org.apache.hadoop.fs.Path;
>import org.apache.hadoop.hbase.HBaseConfiguration;
>import org.apache.hadoop.hbase.KeyValue;
>import org.apache.hadoop.hbase.client.Put;
>import org.apache.hadoop.hbase.client.Result;
>import org.apache.hadoop.hbase.client.Scan;
>import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
>import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;
>import org.apache.hadoop.hbase.mapreduce.TableMapper;
>import org.apache.hadoop.hbase.mapreduce.TableReducer;
>import org.apache.hadoop.hbase.util.Bytes;
>
>import org.apache.hadoop.io.IntWritable;
>import org.apache.hadoop.io.Text;
>import org.apache.hadoop.io.Writable;
>import org.apache.hadoop.mapreduce.Job;
>import org.apache.hadoop.mapreduce.Reducer;
>import org.apache.hadoop.mapreduce.Reducer.Context;
>import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
>import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
>import org.co_occurance.Pair;
>import org.co_occurance.PairsMethod;
>import org.co_occurance.PairsMethod.MeanReducer;
>import org.co_occurance.PairsMethod.PairsMapper;
>
>public class HBaseExperiment {
>public static class MyMapper extends TableMapper  {
>public static final byte[] CF = "cf".getBytes();
>*public static final byte[] ATTR1 = "attr1".getBytes();*
>
>private final IntWritable ONE = new IntWritable(1);
>   private Text text = new Text();
>
>   public void map(ImmutableBytesWritable row, Result value, Context
>context) throws IOException, InterruptedException {
> String val = new String(value.getValue(CF, ATTR1));
>   //text.set(val); // we can only emit Writables...
>   text.set(value.toString());
> context.write(text, ONE);
>   }
>}
> public static class MyTableReducer extends TableReducerIntWritable,
>ImmutableBytesWritable>  {
>public static final byte[] CF = "cf".getBytes();
>public static final byte[] COUNT = "count".getBytes();
>
> public void reduce(Text key, Iterable values, Context
>context) throws IOException, InterruptedException {
> int i = 0;
> for (IntWritable val : values) {
> i += val.get();
> }
> Put put = new Put(Bytes.toBytes(key.toString()));
> put.add(CF, COUNT, Bytes.toBytes(i));
>
> context.write(null, put);
>   }
>}
>
> public static void main(String[] args) throws Exception {
>Configuration config = HBaseConfiguration.create();
>Job job = new Job(config,"ExampleSummary");
>job.setJarByClass(HBaseExperiment.class); // class that contains
>mapper
>and reducer
>

Re: Writing data to hbase from reducer

[Surendra , Manchikanti]

Hbase comes with Bulkload tool. Please check below link.

http://hbase.apache.org/book/arch.bulk.load.html

Regards,
Surendra M


-- Surendra Manchikanti


On Wed, Aug 28, 2013 at 11:39 PM, Doug Meil
wrote:

>
> MapReduce job reading in your data in HDFS and then emitting Puts against
> the target table in the Mapper since it looks like there isn't any
> transform happening...
>
> http://hbase.apache.org/book/mapreduce.example.html
>
> Likewise, what Harsh said a few days ago.
>
> On 8/27/13 6:33 PM, "Harsh J"  wrote:
>
> >You can use HBase's MultiTableOutputFormat:
> >
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/MultiTab
> >leOutputFormat.html
> >
> >An example can be found in this blog post:
> >
> http://www.wildnove.com/2011/07/19/tutorial-hadoop-and-hbase-multitableout
> >putformat/
>
>
>
>
>
>
> On 8/28/13 12:49 PM, "jamal sasha"  wrote:
>
> >Hi,
> >I have data in form:
> >
> >source, destination, connection
> >This data is saved in hdfs
> >
> >I want to read this data and put it in hbase table something like:
> >Column1 (source) | Column2(Destination)| Column3(Connection Type)
> >Rowvertex A|   vertex B | connection
> >
> >How do I do this?
> >Thanks
>
>

Re: Writing data to hbase from reducer

[Doug Meil]

MapReduce job reading in your data in HDFS and then emitting Puts against
the target table in the Mapper since it looks like there isn't any
transform happening...

http://hbase.apache.org/book/mapreduce.example.html
 
Likewise, what Harsh said a few days ago.

On 8/27/13 6:33 PM, "Harsh J"  wrote:

>You can use HBase's MultiTableOutputFormat:
>http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/MultiTab
>leOutputFormat.html
>
>An example can be found in this blog post:
>http://www.wildnove.com/2011/07/19/tutorial-hadoop-and-hbase-multitableout
>putformat/






On 8/28/13 12:49 PM, "jamal sasha"  wrote:

>Hi,
>I have data in form:
>
>source, destination, connection
>This data is saved in hdfs
>
>I want to read this data and put it in hbase table something like:
>Column1 (source) | Column2(Destination)| Column3(Connection Type)
>Rowvertex A|   vertex B | connection
>
>How do I do this?
>Thanks

Writing data to hbase from reducer

[jamal sasha]

Hi,
I have data in form:

source, destination, connection
This data is saved in hdfs

I want to read this data and put it in hbase table something like:
Column1 (source) | Column2(Destination)| Column3(Connection Type)
Rowvertex A|   vertex B | connection

How do I do this?
Thanks

Re: Lease Exception Errors When Running Heavy Map Reduce Job

[Ameya Kanitkar]

Thanks for your response.

I checked namenode logs and I find following:

2013-08-28 15:25:24,025 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: recoverLease: recover
lease [Lease.  Holder:
DFSClient_hb_rs_smartdeals-hbase14-snc1.snc1,60020,1377700014053_-346895658_25,
pendingcreates: 1],
src=/hbase/.logs/smartdeals-hbase14-snc1.snc1,60020,1377700014053-splitting/smartdeals-hbase14-snc1.snc1%2C60020%2C1377700014053.1377700015413
from client
DFSClient_hb_rs_smartdeals-hbase14-snc1.snc1,60020,1377700014053_-346895658_25
2013-08-28 15:25:24,025 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Recovering
lease=[Lease.  Holder:
DFSClient_hb_rs_smartdeals-hbase14-snc1.snc1,60020,1377700014053_-346895658_25,
pendingcreates: 1],
src=/hbase/.logs/smartdeals-hbase14-snc1.snc1,60020,1377700014053-splitting/smartdeals-hbase14-snc1.snc1%2C60020%2C1377700014053.1377700015413
2013-08-28 15:25:24,025 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
internalReleaseLease: All existing blocks are COMPLETE, lease removed, file
closed.

There are LeaseException errors on namenode as well:
http://pastebin.com/4feVcL1F Not sure why its happening.

I do not think I am ending up with any timeouts, as my jobs fail within
couple of minutes, while all my time outs are 10 minutes+
Not sure why above would

Ameya



On Wed, Aug 28, 2013 at 9:00 AM, Ted Yu  wrote:

> From the log you posted on pastebin, I see the following.
> Can you check namenode log to see what went wrong ?
>
>
>1. Caused by:
>org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease
> on
>
>  
> /hbase/.logs/smartdeals-hbase14-snc1.snc1,60020,1376944419197/smartdeals-hbase14-snc1.snc1%2C60020%2C1376944419197.1377699297514
>File does not exist. [Lease.  Holder:
>
>  
> DFSClient_hb_rs_smartdeals-hbase14-snc1.snc1,60020,1376944419197_-413917755_25,
>pendingcreates: 1]
>
>
>
> On Wed, Aug 28, 2013 at 8:00 AM, Ameya Kanitkar  wrote:
>
> > HI All,
> >
> > We have a very heavy map reduce job that goes over entire table with over
> > 1TB+ data in HBase and exports all data (Similar to Export job but with
> > some additional custom code built in) to HDFS.
> >
> > However this job is not very stable, and often times we get following
> error
> > and job fails:
> >
> > org.apache.hadoop.hbase.regionserver.LeaseException:
> > org.apache.hadoop.hbase.regionserver.LeaseException: lease
> > '-4456594242606811626' does not exist
> > at
> > org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2429)
> > at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> > at
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1400)
> >
> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > Method)
> > at
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> > at java.lang.reflect.Constructor.newInstance(Constructor.
> >
> >
> > Here are more detailed logs on the RS: http://pastebin.com/xaHF4ksb
> >
> > We have changed following settings in HBase to counter this problem
> > but issue persists:
> >
> > 
> > 
> > hbase.regionserver.lease.period
> > 90
> > 
> >
> > 
> > 
> > hbase.rpc.timeout
> > 90
> > 
> >
> >
> > We also reduced number of mappers per RS less than available CPU's on the
> > box.
> >
> > We also observed that problem once happens, happens multiple times on
> > the same RS. All other regions are unaffected. But different RS
> > observes this problem on different days. There is no particular region
> > causing this either.
> >
> > We are running: 0.94.2 with cdh4.2.0
> >
> > Any ideas?
> >
> >
> > Ameya
> >
>

Re: Is it possible to get the region count of a table via an API?

[Ted Yu]

You can also use HTable#getRegionLocations():

  public NavigableMap getRegionLocations()
throwsIOException {

FYI


On Wed, Aug 28, 2013 at 6:12 AM, Surendra , Manchikanti <
surendra.manchika...@gmail.com> wrote:

> List<
> http://docs.oracle.com/javase/6/docs/api/java/util/List.html?is-external=true
> >
>  http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HRegionInfo.html>>
> = HBaseAdmin#getTableRegions(tablename);
>
> HRegionInfo.getServerName();
>
> Regards,
> Surendra M
>
> -- Surendra Manchikanti
>
>
> On Wed, Aug 28, 2013 at 5:59 PM, Pavan Sudheendra  >wrote:
>
> > How do i get the Server Name associated with the region?
> >
> >
> > On Wed, Aug 28, 2013 at 3:46 PM, Ashwanth Kumar <
> > ashwanthku...@googlemail.com> wrote:
> >
> > > To check how regions you have in a table (and possibly what they are)
> > >  HBaseAdmin#getTableRegions<
> > >
> >
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#getTableRegions(byte[])
> > > >.
> > > In order to split the table you can use
> > > HBAdmin#split<
> > >
> >
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#split(byte[])
> > > >
> > > .
> > >
> > > PS: #split() is an Async Operation.
> > >
> > >
> > > On Wed, Aug 28, 2013 at 3:41 PM, Pavan Sudheendra  > > >wrote:
> > >
> > > > Hi all,
> > > > I know what we can go over to the HBase UI and make a split on our
> > table
> > > so
> > > > that it will be distributed over the cluster.. Is there a way to know
> > it
> > > > via an API and to possibly change it? This is to know how many map
> > tasks
> > > > run on our table before we actually run the MR job..
> > > > --
> > > > Regards-
> > > > Pavan
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Ashwanth Kumar / ashwanthkumar.in
> > >
> >
> >
> >
> > --
> > Regards-
> > Pavan
> >
>

Re: Lease Exception Errors When Running Heavy Map Reduce Job

[Ted Yu]

>From the log you posted on pastebin, I see the following.
Can you check namenode log to see what went wrong ?


   1. Caused by:
   org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
   
/hbase/.logs/smartdeals-hbase14-snc1.snc1,60020,1376944419197/smartdeals-hbase14-snc1.snc1%2C60020%2C1376944419197.1377699297514
   File does not exist. [Lease.  Holder:
   
DFSClient_hb_rs_smartdeals-hbase14-snc1.snc1,60020,1376944419197_-413917755_25,
   pendingcreates: 1]



On Wed, Aug 28, 2013 at 8:00 AM, Ameya Kanitkar  wrote:

> HI All,
>
> We have a very heavy map reduce job that goes over entire table with over
> 1TB+ data in HBase and exports all data (Similar to Export job but with
> some additional custom code built in) to HDFS.
>
> However this job is not very stable, and often times we get following error
> and job fails:
>
> org.apache.hadoop.hbase.regionserver.LeaseException:
> org.apache.hadoop.hbase.regionserver.LeaseException: lease
> '-4456594242606811626' does not exist
> at
> org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2429)
> at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1400)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.
>
>
> Here are more detailed logs on the RS: http://pastebin.com/xaHF4ksb
>
> We have changed following settings in HBase to counter this problem
> but issue persists:
>
> 
> 
> hbase.regionserver.lease.period
> 90
> 
>
> 
> 
> hbase.rpc.timeout
> 90
> 
>
>
> We also reduced number of mappers per RS less than available CPU's on the
> box.
>
> We also observed that problem once happens, happens multiple times on
> the same RS. All other regions are unaffected. But different RS
> observes this problem on different days. There is no particular region
> causing this either.
>
> We are running: 0.94.2 with cdh4.2.0
>
> Any ideas?
>
>
> Ameya
>

Re: Lease Exception Errors When Running Heavy Map Reduce Job

[Dhaval Shah]

Couple of things:
- Can you check the resources on the region server for which you get the lease 
exception? It seems like the server is heavily thrashed
- What are your values for scan.setCaching and scan.setBatch? 



The lease does not exist exception generally happens when the client goes back 
to the region server after the lease expires (in your case 90). If you 
setCaching is really high for example, the client gets enough data in one call 
to scanner.next and keeps processing it for > 90 ms and when it eventually 
goes back to the region server, the lease on the region server has already 
expired. Setting your setCaching value lower might help in this case

Regards,
Dhaval



From: Ameya Kanitkar 
To: user@hbase.apache.org 
Sent: Wednesday, 28 August 2013 11:00 AM
Subject: Lease Exception Errors When Running Heavy Map Reduce Job


HI All,

We have a very heavy map reduce job that goes over entire table with over
1TB+ data in HBase and exports all data (Similar to Export job but with
some additional custom code built in) to HDFS.

However this job is not very stable, and often times we get following error
and job fails:

org.apache.hadoop.hbase.regionserver.LeaseException:
org.apache.hadoop.hbase.regionserver.LeaseException: lease
'-4456594242606811626' does not exist
    at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231)
    at 
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2429)
    at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
    at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at 
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
    at 
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1400)

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.


Here are more detailed logs on the RS: http://pastebin.com/xaHF4ksb

We have changed following settings in HBase to counter this problem
but issue persists:



hbase.regionserver.lease.period
90




hbase.rpc.timeout
90



We also reduced number of mappers per RS less than available CPU's on the box.

We also observed that problem once happens, happens multiple times on
the same RS. All other regions are unaffected. But different RS
observes this problem on different days. There is no particular region
causing this either.

We are running: 0.94.2 with cdh4.2.0

Any ideas?


Ameya 

How to enable metrics2?

[Ionut Ignatescu]

Hi,

I am using HBase 0.94.11 with Hadoop 1.1.2
I want to improve my current monitoring solution and I create a custom
MetricsSink that export metrics in a custom format. This solution runs
perfect with Hadoop.
Unfortunately, I cannot say the same thing about HBase.
I have several questions:
1. What is the correct name for properties file?
I found various examples with hadoop-metrics2.properties and several jira
items (6405, 6408) that say file name should be hadoop-hbase...
2. Is there any log entry reporting metric was enabled and if yes, where is
that printed? I saw hadoop has an explicit entry saying it found or not
properties file.
3. Is there any config parameter that disables metrics2?

Thanks!

Lease Exception Errors When Running Heavy Map Reduce Job

[Ameya Kanitkar]

HI All,

We have a very heavy map reduce job that goes over entire table with over
1TB+ data in HBase and exports all data (Similar to Export job but with
some additional custom code built in) to HDFS.

However this job is not very stable, and often times we get following error
and job fails:

org.apache.hadoop.hbase.regionserver.LeaseException:
org.apache.hadoop.hbase.regionserver.LeaseException: lease
'-4456594242606811626' does not exist
at 
org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231)
at 
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2429)
at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at 
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1400)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.


Here are more detailed logs on the RS: http://pastebin.com/xaHF4ksb

We have changed following settings in HBase to counter this problem
but issue persists:



hbase.regionserver.lease.period
90




hbase.rpc.timeout
90



We also reduced number of mappers per RS less than available CPU's on the box.

We also observed that problem once happens, happens multiple times on
the same RS. All other regions are unaffected. But different RS
observes this problem on different days. There is no particular region
causing this either.

We are running: 0.94.2 with cdh4.2.0

Any ideas?


Ameya

Re: how to export data from hbase to mysql?

[Shahab Yunus]

Taking what Ravi Kiran mentioned a level higher, you can also use Pig. It
has DBStorage. Very easy to rad from HBase and dump to MySQL if your data
porting does not require complex transformation (even which can be handled
in Pig too.)

http://pig.apache.org/docs/r0.11.0/api/org/apache/pig/piggybank/storage/DBStorage.html

Regards,
Shahab


On Wed, Aug 28, 2013 at 1:26 AM, Ravi Kiran wrote:

> If you would like to have greater control on what data / which columns
> from HBase should be going into MySQL tables , you can write a simple MR
> job and use the DBOutputFormat .   It is a simple one and works great for
> us.
>
> Regards
> Ravi
>
>
>
> On Wed, Aug 28, 2013 at 10:42 AM, ch huang  wrote:
>
> > i use hive ,maybe it's a way ,let me try  it
> >
> > On Wed, Aug 28, 2013 at 11:21 AM, James Taylor  > >wrote:
> >
> > > Or if you'd like to be able to use SQL directly on it, take a look at
> > > Phoenix (https://github.com/forcedotcom/phoenix).
> > >
> > > James
> > >
> > > On Aug 27, 2013, at 8:14 PM, Jean-Marc Spaggiari
> > >   wrote:
> > >
> > > > Take a look at sqoop?
> > > > Le 2013-08-27 23:08, "ch huang"  a écrit :
> > > >
> > > >> hi,all:
> > > >> any good idea? thanks
> > > >>
> > >
> >
>

Re: Re: Will hbase automatically distribute the data across region servers or NOT..??

[Frank Chow]

Hi,

According the error message in the master log, there maybe some inconsistencies 
in the configuration, check the configuration on all nodes, if the properity 
below is configured, and if it's inconsistent.

  
hbase.metrics.showTableName
true
Whether to include the prefix "tbl.tablename" in per-column 
family metrics.
If true, for each metric M, per-cf metrics will be reported for tbl.T.cf.CF.M, 
if false,
per-cf metrics will be aggregated by column-family across tables, and reported 
for cf.CF.M.
In both cases, the aggregated metric M across tables and cfs will be reported.

  




Frank Chow

Re: Is it possible to get the region count of a table via an API?

[Surendra , Manchikanti]

List
http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HRegionInfo.html>>
= HBaseAdmin#getTableRegions(tablename);

HRegionInfo.getServerName();

Regards,
Surendra M

-- Surendra Manchikanti


On Wed, Aug 28, 2013 at 5:59 PM, Pavan Sudheendra wrote:

> How do i get the Server Name associated with the region?
>
>
> On Wed, Aug 28, 2013 at 3:46 PM, Ashwanth Kumar <
> ashwanthku...@googlemail.com> wrote:
>
> > To check how regions you have in a table (and possibly what they are)
> >  HBaseAdmin#getTableRegions<
> >
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#getTableRegions(byte[])
> > >.
> > In order to split the table you can use
> > HBAdmin#split<
> >
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#split(byte[])
> > >
> > .
> >
> > PS: #split() is an Async Operation.
> >
> >
> > On Wed, Aug 28, 2013 at 3:41 PM, Pavan Sudheendra  > >wrote:
> >
> > > Hi all,
> > > I know what we can go over to the HBase UI and make a split on our
> table
> > so
> > > that it will be distributed over the cluster.. Is there a way to know
> it
> > > via an API and to possibly change it? This is to know how many map
> tasks
> > > run on our table before we actually run the MR job..
> > > --
> > > Regards-
> > > Pavan
> > >
> >
> >
> >
> > --
> >
> > Ashwanth Kumar / ashwanthkumar.in
> >
>
>
>
> --
> Regards-
> Pavan
>

Re: Is it possible to get the region count of a table via an API?

[Pavan Sudheendra]

How do i get the Server Name associated with the region?


On Wed, Aug 28, 2013 at 3:46 PM, Ashwanth Kumar <
ashwanthku...@googlemail.com> wrote:

> To check how regions you have in a table (and possibly what they are)
>  HBaseAdmin#getTableRegions<
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#getTableRegions(byte[])
> >.
> In order to split the table you can use
> HBAdmin#split<
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#split(byte[])
> >
> .
>
> PS: #split() is an Async Operation.
>
>
> On Wed, Aug 28, 2013 at 3:41 PM, Pavan Sudheendra  >wrote:
>
> > Hi all,
> > I know what we can go over to the HBase UI and make a split on our table
> so
> > that it will be distributed over the cluster.. Is there a way to know it
> > via an API and to possibly change it? This is to know how many map tasks
> > run on our table before we actually run the MR job..
> > --
> > Regards-
> > Pavan
> >
>
>
>
> --
>
> Ashwanth Kumar / ashwanthkumar.in
>



-- 
Regards-
Pavan

experiencing high latency for few reads in HBase

[Saurabh Yahoo]

Hi,

We are running a stress test in our 5 node cluster and we are getting the 
expected mean latency of 10ms. But we are seeing around 20 reads out of 25 
million reads having latency more than 4 seconds. Can anyone provide the 
insight what we can do to meet below second SLA for each and every read?

We observe the following things - 

1. Reads are evenly distributed among 5 nodes.  CPUs remain under 5% utilized. 

2. We have 4gb block cache (30% block cache out of 12gb) setup. 3gb block cache 
got filled up but around 1gb remained free. There are a large number of cache 
eviction.

Questions to experts -

1. If there are still 1gb of free block cache available, why is hbase evicting 
the block from cache? 

4. We are seeing memory went up to 10gb three times before dropping sharply to 
5gb. 

Any help is highly appreciable,

Thanks,
Saurabh. 

Re: Is it possible to get the region count of a table via an API?

[Ashwanth Kumar]

To check how regions you have in a table (and possibly what they are)
 
HBaseAdmin#getTableRegions.
In order to split the table you can use
HBAdmin#split
.

PS: #split() is an Async Operation.


On Wed, Aug 28, 2013 at 3:41 PM, Pavan Sudheendra wrote:

> Hi all,
> I know what we can go over to the HBase UI and make a split on our table so
> that it will be distributed over the cluster.. Is there a way to know it
> via an API and to possibly change it? This is to know how many map tasks
> run on our table before we actually run the MR job..
> --
> Regards-
> Pavan
>



-- 

Ashwanth Kumar / ashwanthkumar.in

Is it possible to get the region count of a table via an API?

[Pavan Sudheendra]

Hi all,
I know what we can go over to the HBase UI and make a split on our table so
that it will be distributed over the cluster.. Is there a way to know it
via an API and to possibly change it? This is to know how many map tasks
run on our table before we actually run the MR job..
-- 
Regards-
Pavan

Re: Data Deduplication in HBase

[Anand Nalya]

Hi Chris,

Thanks a lot for the detailed response. I'll definitely try this design and
see how it performs.

Anand


On 28 August 2013 13:56, Chris Perluss  wrote:

> It might help to pick a granularity level. For example let's suppose you
> pick a granularity level of 0.1.
>
> Any piece of the song you receive should be broken down into segments of
> 0.1 and they need to be aligned on 0.1.
>
> Example: you receive a piece of the song from 0.65 to 0.85.
> You would break this into three segments:
> 0.65 to 0.70
> 0.70 to 0.80
> 0.80.to 0.85
>
> These three segments would get written to three different rows.  The row
> key would be the song identifier followed by the segment number.  The first
> row would be "songId-0.6", the second "songId-0.7" , and the third
> "songId-0.8".
>
> The first row is "songId-0.6" and not "songId-0.65" because you want all
> pieces of the song between 0.6 and 0.7 to end up in the same row.  You do
> this by rounding down to the start range for the segment.
>
> When writing the three example segments to HBase there will be two
> scenarios.
>
> The first scenario is that you have an entire segment to be saved. In the
> above example this is the case for your piece that spans the 0.7 to 0.8
> segment.  Since you have the entire segment you don't have to combine it
> with any existing data.  So you can simply do a put and overwrite any
> partial data that might happen to exist in that row.  If you configure your
> column family to only store one version for each cell then this will
> perform "deduping" for that segment because it will only keep your new,
> complete version of that segment.
>
> The other scenario is that you receive a part of a segment.  In this case
> you will need to read in the row corresponding to your segment, combine
> your new partial segment with any existing partial segment, then put the
> combined segment back into hbase.
> In the above example this applies to the 0.65 to 0.7 segment (and the 0.8
> to 0.85 segment).
>
> When you read the row at "songId-0.6",  if there is already data there you
> will need to combine it with your new data.  E.g. if you found 0.63 to 0.67
> you would combine it with 0.65 to 0.70 and end up with 0.63 to 0.70.  Then
> write this segment back to hbase. If you have versions set to 1 then this
> bigger segment will replace the smaller segment you had before, thus
> "deduping" that particular segment.
>
> If you think overlapping segments will potentially be uploaded at the same
> time then you will need to implement an optimistic locking model using
> checkAndPut.  I would do this by defining one column to contain the song
> data and another column to contain a row version.  I can go into more
> detail if requested.
>
> Here's the benefit of this design:
> 1.  Each row will have approximately the same size (KB of song data).  E.g.
> you don't have to worry about someone uploading a 2 hour long epic folk
> metal song (I'm looking at you, Moonsorrow!) and thus creating a cell too
> big for hbase to efficiently handle.  This 2 hour long song will be broken
> up over lots of rows.
> 2.  You can tune the row size by changing the granularity (before you go
> into production!)
> 3.  For each upload request you will only need to Get a max of two segments
> from HBase in order to append to a partial segment.  The only segments you
> need to Get would be the partial segment at the beginning of the upload and
> the partial segment at the end of the upload.  All segments between these
> two are complete segments and thus can just Put their entire contents into
> the right row.
> 4.  Since you only Get two segments you will only read in a few 100 kbs of
> data in order to perform the update (amount read in depends on your
> granularity). This is true no matter how much of the file has already been
> uploaded.  In a non segmented storage scenario where you stored the entire
> file in one cell, if 30 MB had already been uploaded then a request to
> upload an additional 100 KB would require reading in all 30MB and writing
> back all 30.1 MB back to HBase.
> 5. You can easily and efficiently retrieve a completed song by performing a
> Scan using the songId.  Ie, Scan(rowStart="songId-", rowEnd="songId.")
> "." Is the next ascii char after "-".
>
> Hope this helps!
>  On Aug 27, 2013 10:13 PM, "Anand Nalya"  wrote:
>
> > The segments are completely random. The segments can have from no overlap
> > to exact duplicates.
> >
> > Anand
> >
> >
> > On 27 August 2013 19:49, Ted Yu  wrote:
> >
> > > bq.  Will hbase do some sort of deduplication?
> > >
> > > I don't think so.
> > >
> > > What is the granularity of segment overlap ? In the above example, it
> > seems
> > > to be 0.5
> > >
> > > Cheers
> > >
> > >
> > > On Tue, Aug 27, 2013 at 7:12 AM, Anand Nalya 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I have a use case in which I need to store segments of mp3 files in
> > > hbase.
> > > > A song may come to the application in different ovelapping 

Re: Hbase 0.94.6 stargate can't use multi get

[Dmitriy Troyan]

Hey Ravi,
Seems I find what problem was: when I communicate with stargate I not set
Accept header to application/json. It was octet-stream and according to
documentation it can only give one value.

Thanks.


On Wed, Aug 28, 2013 at 8:46 AM, Dmitriy Troyan wrote:

> Please excuse my inattention. I made mistake in my query, it need to be
> like this
>
> http://myhost:8080/updaterlog/12345/data:log/0,1377598680245/?v=100
>
> In my message I didn't write row name in query (12345).
>
> *But still I have the same problem - only one row return from stargate.*
>
> Thanks.
>
>
>
> On Wed, Aug 28, 2013 at 8:37 AM, Dmitriy Troyan 
> wrote:
>
>> Hi, Ravi
>> This is my schema:
>>
>> { NAME=> 'updaterlog', IS_META => 'false', IS_ROOT => 'false', COLUMNS => [ 
>> { NAME => 'data', BLOCKSIZE => '65536', BLOOMFILTER => 'NONE', MIN_VERSIONS 
>> => '0', KEEP_DELETED_CELLS => 'false', ENCODE_ON_DISK => 'true', BLOCKCACHE 
>> => 'true', COMPRESSION => 'NONE', VERSIONS => '100', REPLICATION_SCOPE => 
>> '0', TTL => '2147483647', DATA_BLOCK_ENCODING => 'NONE', IN_MEMORY => 
>> 'false' } ] }
>>
>> This is another table but result of query like 
>> > > is same - always 
>> the only one result.
>>
>> This is the result in hbase shell (maybe it help to understand this 
>> problem): http://img843.imageshack.us/i/rx0g.png/
>>
>> Thanks
>>
>>
>>
>>
>> On Wed, Aug 28, 2013 at 7:45 AM, Ravi Kiran wrote:
>>
>>> Hi ,
>>> Can you please query for the schema of the table and show us here.
>>> Would like to know what is value for VERSIONS that you have set for the
>>> column family .  I hope you have set it to 10.
>>>
>>> Ex:http://myhost.com:8080/log/schema<
>>> http://myhost.com:8080/log/data:get/0,1377633354/?v=10>
>>>
>>>
>>> Regards
>>> Ravi Magham
>>>
>>>
>>> On Wed, Aug 28, 2013 at 1:29 AM, Dmitriy Troyan <
>>> troyan.dmit...@gmail.com>wrote:
>>>
>>> > Hey all,
>>> >
>>> > I try to use multi get for receiving different versions of row but it
>>> give
>>> > me only one always. For example I have table log, and column family
>>> > data:get. I put a lot of versions of row/data:log. Now I try to get all
>>> > versions of this key.
>>> >
>>> > As it said in manual http://wiki.apache.org/hadoop/Hbase/Stargate(Cell or
>>> > Row Query (Multiple Values)): using browser for request
>>> > http://myhost.com:8080/log/data:get/0,1377633354/?v=10
>>> >
>>> > The response contain only one version of a key instid of give me all
>>> (max
>>> > 10) available versions of a key.
>>> >
>>> > I break my brain on this problem. Please reffer me to right way.
>>> >
>>> > Thanks!
>>> >
>>>
>>
>>
>

Re: hadoop2 and Hbase0.94

[Harsh J]

Moving to user@hbase.apache.org.

Please share your hbase-site.xml and core-site.xml. Was this HBase
cluster previously running on a standalone local filesystem mode?

On Wed, Aug 28, 2013 at 2:06 PM, lei liu  wrote:
> I use hadoop2 and hbase0.94, but there is below exception:
>
> 2013-08-28 11:36:12,922 ERROR
> [MASTER_TABLE_OPERATIONS-dw74.kgb.sqa.cm4,13646,1377660964832-0]
> executor.EventHandler(172): Caught throwable while processing
> event C_M_DELETE_TABLE
> java.lang.IllegalArgumentException: Wrong FS:
> file:/tmp/hbase-shenxiu.cx/hbase/observed_table/47b334989065a8ac84873e6d07c1de62,
> expected: hdfs://localhost.lo
> caldomain:35974
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:590)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:172)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:402)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1427)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1467)
> at
> org.apache.hadoop.hbase.util.FSUtils.listStatus(FSUtils.java:1052)
> at
> org.apache.hadoop.hbase.backup.HFileArchiver.archiveRegion(HFileArchiver.java:123)
> at
> org.apache.hadoop.hbase.backup.HFileArchiver.archiveRegion(HFileArchiver.java:72)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.deleteRegion(MasterFileSystem.java:444)
> at
> org.apache.hadoop.hbase.master.handler.DeleteTableHandler.handleTableOperation(DeleteTableHandler.java:73)
> at
> org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:96)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 2013-08-28 11:37:05,653 INFO
> [Master:0;dw74.kgb.sqa.cm4,13646,1377660964832.archivedHFileCleaner]
> util.FSUtils(1055): hdfs://localhost.localdomain:35974/use



-- 
Harsh J

Re: Data Deduplication in HBase

[Chris Perluss]

It might help to pick a granularity level. For example let's suppose you
pick a granularity level of 0.1.

Any piece of the song you receive should be broken down into segments of
0.1 and they need to be aligned on 0.1.

Example: you receive a piece of the song from 0.65 to 0.85.
You would break this into three segments:
0.65 to 0.70
0.70 to 0.80
0.80.to 0.85

These three segments would get written to three different rows.  The row
key would be the song identifier followed by the segment number.  The first
row would be "songId-0.6", the second "songId-0.7" , and the third
"songId-0.8".

The first row is "songId-0.6" and not "songId-0.65" because you want all
pieces of the song between 0.6 and 0.7 to end up in the same row.  You do
this by rounding down to the start range for the segment.

When writing the three example segments to HBase there will be two
scenarios.

The first scenario is that you have an entire segment to be saved. In the
above example this is the case for your piece that spans the 0.7 to 0.8
segment.  Since you have the entire segment you don't have to combine it
with any existing data.  So you can simply do a put and overwrite any
partial data that might happen to exist in that row.  If you configure your
column family to only store one version for each cell then this will
perform "deduping" for that segment because it will only keep your new,
complete version of that segment.

The other scenario is that you receive a part of a segment.  In this case
you will need to read in the row corresponding to your segment, combine
your new partial segment with any existing partial segment, then put the
combined segment back into hbase.
In the above example this applies to the 0.65 to 0.7 segment (and the 0.8
to 0.85 segment).

When you read the row at "songId-0.6",  if there is already data there you
will need to combine it with your new data.  E.g. if you found 0.63 to 0.67
you would combine it with 0.65 to 0.70 and end up with 0.63 to 0.70.  Then
write this segment back to hbase. If you have versions set to 1 then this
bigger segment will replace the smaller segment you had before, thus
"deduping" that particular segment.

If you think overlapping segments will potentially be uploaded at the same
time then you will need to implement an optimistic locking model using
checkAndPut.  I would do this by defining one column to contain the song
data and another column to contain a row version.  I can go into more
detail if requested.

Here's the benefit of this design:
1.  Each row will have approximately the same size (KB of song data).  E.g.
you don't have to worry about someone uploading a 2 hour long epic folk
metal song (I'm looking at you, Moonsorrow!) and thus creating a cell too
big for hbase to efficiently handle.  This 2 hour long song will be broken
up over lots of rows.
2.  You can tune the row size by changing the granularity (before you go
into production!)
3.  For each upload request you will only need to Get a max of two segments
from HBase in order to append to a partial segment.  The only segments you
need to Get would be the partial segment at the beginning of the upload and
the partial segment at the end of the upload.  All segments between these
two are complete segments and thus can just Put their entire contents into
the right row.
4.  Since you only Get two segments you will only read in a few 100 kbs of
data in order to perform the update (amount read in depends on your
granularity). This is true no matter how much of the file has already been
uploaded.  In a non segmented storage scenario where you stored the entire
file in one cell, if 30 MB had already been uploaded then a request to
upload an additional 100 KB would require reading in all 30MB and writing
back all 30.1 MB back to HBase.
5. You can easily and efficiently retrieve a completed song by performing a
Scan using the songId.  Ie, Scan(rowStart="songId-", rowEnd="songId.")
"." Is the next ascii char after "-".

Hope this helps!
 On Aug 27, 2013 10:13 PM, "Anand Nalya"  wrote:

> The segments are completely random. The segments can have from no overlap
> to exact duplicates.
>
> Anand
>
>
> On 27 August 2013 19:49, Ted Yu  wrote:
>
> > bq.  Will hbase do some sort of deduplication?
> >
> > I don't think so.
> >
> > What is the granularity of segment overlap ? In the above example, it
> seems
> > to be 0.5
> >
> > Cheers
> >
> >
> > On Tue, Aug 27, 2013 at 7:12 AM, Anand Nalya 
> > wrote:
> >
> > > Hi,
> > >
> > > I have a use case in which I need to store segments of mp3 files in
> > hbase.
> > > A song may come to the application in different ovelapping segments.
> For
> > > example, a 5 min song can have the following segments
> 0-1,0.5-2,2-4,3-5.
> > As
> > > seen, some of the data is duplicate (3-4 is present in the last 2
> > > segments).
> > >
> > > What would be the ideal way of removing this duplicate storage? Will
> > snappy
> > > compression help here or do I need to write some logic over HBase?
> Also,
> >