RE: Scan performance

[Vladimir Rodionov]

Lars,
I thought that column family is the locality group and placement columns which 
are frequently accessed together into
the same column family (locality group) is the obvious performance improvement 
tip. What are the "essential column families" for in this context?

As for original question..  Unless you place your column into a separate column 
family in Table 2, you will
need to scan (load from disk if not cached) ~ 40x more data for the second 
table (because you have 40 columns). This may explain why do  see such a 
difference in
execution time if all data needs to be loaded first from HDFS.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: lars hofhansl [la...@apache.org]
Sent: Friday, June 21, 2013 3:37 PM
To: user@hbase.apache.org
Subject: Re: Scan performance

HBase is a key value (KV) store. Each column is stored in its own KV, a row is 
just a set of KVs that happen to have the row key (which is the first part of 
the key).
I tried to summarize this here: 
http://hadoop-hbase.blogspot.de/2011/12/introduction-to-hbase.html)

In the StoreFiles all KVs are sorted in row/column order, but HBase still needs 
to skip over many KVs in order to "reach" the next row. So more disk and memory 
IO is needed.

If you using 0.94 there is a new feature "essential column families". If you 
always search by the same column you can place that one in its own column 
family and all other column in another column family. In that case your scan 
performance should be close identical.


-- Lars


From: Tony Dean 
To: "user@hbase.apache.org" 
Sent: Friday, June 21, 2013 2:08 PM
Subject: Scan performance




Hi,

I hope that you can shed some light on these 2 scenarios below.

I have 2 small tables of 6000 rows.
Table 1 has only 1 column in each of its rows.
Table 2 has 40 columns in each of its rows.
Other than that the two tables are identical.

In both tables there is only 1 row that contains a matching column that I am 
filtering on.   And the Scan performs correctly in both cases by returning only 
the single result.

The code looks something like the following:

Scan scan = new Scan(startRow, stopRow);   // the start/stop rows should 
include all 6000 rows
scan.addColumn(cf, qualifier); // only return the column that I am interested 
in (should only be in 1 row and only 1 version)

Filter f1 = new InclusiveStopFilter(stopRow);
Filter f2 = new SingleColumnValueFilter(cf, qualifier,  
CompareFilter.CompareOp.EQUALS, value);
scan.setFilter(new FilterList(f1, f2));

scan .setTimeRange(0, MAX_LONG);
scan.setMaxVersions(1);

ResultScanner rs = t.getScanner(scan);
for (Result result: rs)
{

}

For table 1, rs.next() takes about 30ms.
For table 2, rs.next() takes about 180ms.

Both are returning the exact same result.  Why is it taking so much longer on 
table 2 to get the same result?  The scan depth is the same.  The only 
difference is the column width.  But I’m filtering on a single column and 
returning only that column.

Am I missing something?  As I increase the number of columns, the response time 
gets worse.  I do expect the response time to get worse when increasing the 
number of rows, but not by increasing the number of columns since I’m returning 
only 1 column in
both cases.

I appreciate any comments that you have.

-Tony



Tony Dean
SAS Institute Inc.
Principal Software Developer
919-531-6704  …

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: difference between major and minor compactions?

[Vladimir Rodionov]

Major compactions floods network, leaving for other operations too little 
space. The reason why major compaction are
so prohibitively expensive in HBase - 2 block replicas which need to be created 
in the cluster for every block written locally.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Suraj Varma [svarma...@gmail.com]
Sent: Saturday, June 22, 2013 11:51 AM
To: user@hbase.apache.org
Subject: Re: difference between major and minor compactions?

> In contrast, the major compaction is invoked in  offpeak time and usually
> can be assume to have resource exclusively.

There is no resource exclusivity with major compactions. It is just more
resource _intensive_ because a major compaction will rewrite all the store
files to end up with a single store file per store as described in 9.7.6.5
Compaction in the hbase book.

So - it is because it is so resource _intensive_ that for large clusters
folks prefer to have a managed compaction (i.e. turn off major compaction
and run it off hours) so that it doesn't affect latencies for low latency
consumers, for instance.
--S



On Sat, Jun 22, 2013 at 7:35 AM, yun peng  wrote:

> I am more concerned with CompactionPolicy available that allows application
> to manipulate a bit how compaction should go... It looks like  there is
> newest API in .97 version
> *ExploringCompactionPolicy*<
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/regionserver/compactions/ExploringCompactionPolicy.html
> >,
> which allow application when we should have a major compaction.
>
> For stripe compaction, it is very interesting, will look into it. Thanks.
> Yun
>
>
> On Sat, Jun 22, 2013 at 9:24 AM, Jean-Marc Spaggiari <
> jean-m...@spaggiari.org> wrote:
>
> > Hi Yun,
> >
> > There is more differences.
> >
> > The minor compactions are not remove the delete flags and the deleted
> > cells. It only merge the small files into a bigger one. Only the major
> > compaction (in 0.94) will deal with the delete cells. There is also
> > some more compaction mechanism coming in trunk with nice features.
> >
> > Look at: https://issues.apache.org/jira/browse/HBASE-7902
> > https://issues.apache.org/jira/browse/HBASE-7680
> > https://issues.apache.org/jira/browse/HBASE-7680
> >
> > Minor compactions are promoted to major compactions when the
> > compaction policy decide to compact all the files. If all the files
> > need to be merged, then we can run a major compaction which will do
> > the same thing as the minor one, but with the bonus of deleting the
> > required marked cells.
> >
> > JM
> >
> > 2013/6/22 yun peng :
> > > Thanks, JM
> > > It seems like the sole difference btwn major and minor compaction is
> the
> > > number of files (to be all or just a subset of storefiles). It
> mentioned
> > > very briefly in
> > > http://hbase.apache.org/book<
> > http://hbase.apache.org/book/regions.arch.html>that
> > > "Sometimes a minor compaction will ... promote itself to being a major
> > > compaction". What does "sometime" exactly mean here? or any policy in
> > HBase
> > > that allow application to specify when to promote a minor compaction to
> > be
> > > a major (like user or some monitoring service can specify now is
> offpeak
> > > time?)
> > > Yun
> > >
> > >
> > >
> > > On Sat, Jun 22, 2013 at 8:51 AM, Jean-Marc Spaggiari <
> > > jean-m...@spaggiari.org> wrote:
> > >
> > >> Hi Yun,
> > >>
> > >> Few links:
> > >> - http://blog.cloudera.com/blog/2012/06/hbase-io-hfile-input-output/
> > >> => There is a small paragraph about compactions which explain when
> > >> they are triggered.
> > >> - http://hbase.apache.org/book/regions.arch.html 9.7.6.5
> > >>
> > >> You are almost right. Only thing is that HBase doesn't know when is
> > >> your offpeak, so a major compaction can be triggered anytime if the
> > >> minor is promoted to be a major one.
> > >>
> > >> JM
> > >>
> > >> 2013/6/22 yun peng :
> > >> > Hi, All
> > >> >
> > >> > I am asking the different practices of major and minor compaction...
> > My
> > >> > current understanding is that minor compaction, triggered
> > automatically,
> > >> > usually run along with online query serving (but in background), so
> > that
> &

RE: Hbase random read performance

[Vladimir Rodionov]

Boris if your working set does not fit comfortably into 0.4 * 9 * 12GB ~ 43GB 
HBase cluster block cache ( and I suppose
it does not, because it is less than 1% of your data set  ) - you will be 
depended on your local data nodes  (OS page cache + raw disk I/O) performance.
Your maximum raw I/O is approx 150  (one HDD) * 2 (per HDDsnodes) * 9 (DN #) = 
2700 IOPS - this is what you can get in theory. You are getting 800-1000
- which is not bad taking into account that short circuit reads are probably 
disabled and even not supported in your release.

1. Upgrade to the latest CDH 4.3 if possible
2. Enable short circuit reads in DNs
3. Increase number of nodes (and decrease data set size per node)
4. Increase number of disks per node
5. Increase RS heap memory and allocate more for block caches.
6. major compact critical tables to ensure RS data locality

You need to fit your working set into OS page cache  (or into block cache) to 
get decent random IO performance


Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Ted Yu [yuzhih...@gmail.com]
Sent: Monday, July 08, 2013 5:49 AM
To: user@hbase.apache.org
Subject: Re: Hbase random read performance

Moving to HBase user mailing list.

Can you upgrade to newer release such as 0.94.8 ?

Cheers

On Jul 8, 2013, at 4:36 AM, Boris Emelyanov  wrote:

> I'm trying to configure hbase for fully random read performance, my cluster 
> parameters are:
>
> 9 servers as slaves, each has two 1TB HDD as hadoop volumes;
> data:  800 millions 6-10 KB objects in hbase;
>
> HBase Version - 0.90.6-cdh3u5
> HBASE_HEAPSIZE=12288;
> hfile.block.cache.size = 0.4
> hfile.min.blocksize.size = 16384
> hbase.regionserver.handler.count = 100
>
> I used several recommended tuning solutions, such as
> * tunrning on bloom filters;
> * decreasing hbase block size to 16384.
>
> However, read performance is still poor, about 800-1000rps.
>
> What would you recommend to solve the problem?
>
> --
> Best regards,
>
> Boris.
>

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: optimizing block cache requests + eviction

[Vladimir Rodionov]

Viral,

>From what you described here I can conclude that either:

1. your data access pattern is random and uniform (no data locality at all)
2. or you trash block cache with scan operations with block cache enabled.

or both.

but the idea of treating local and remote blocks differently is very good. +1.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Viral Bajaria [viral.baja...@gmail.com]
Sent: Monday, July 08, 2013 3:04 AM
To: user@hbase.apache.org
Subject: optimizing block cache requests + eviction

Hi,

TL;DR;
Trying to make a case for making the block eviction strategy smart and to
not evict remote blocks more frequently and make the requests more smarter.

The question here comes after I debugged the issue that I was having with
random region servers hitting high load averages. I initially thought the
problem was hardware related i.e. bad disk or network since the wait I/O
was too high but it was a combination of things.

I figured with SCR (short circuit read) ON the datanode should almost never
show high amount of block requests from the local regionservers. So my
starting point for debugging was the datanode since it was doing a ton of
I/O. The clienttrace logs helped me figure out which RS nodes were making
block requests. I hacked up a script to report which blocks are being
requested and how many times per minute. I found that some blocks were
being requested 10+ times in a minute and over 2000 times in an hour from
the same regionserver. This was causing the server to do 40+MB/s on reads
alone. That was on the higher side, the average was closer to 100 or so per
hour.

Now why did I end up in such a situation. It happened due to the fact that
I added servers to the cluster and rebalanced the cluster. At the same time
I added some drives and also removed the offending server in my setup. This
caused some of the data to not be co-located with the regionservers. Given
that major_compaction was disabled and it would not have run for a while
(atleast on some tables) these block requests would not go away. One of my
regionservers was totally overwhelmed. I made the situation worse when I
removed the server that was under heavy load with the assumption that it's
a hardware problem with the box without doing a deep dive (doh!). Given
that regionservers will be added in the future, I expect block locality to
go down till major_compaction runs. Also nodes can go down and cause this
problem. So I started thinking of probable solutions, but first some
observations.

*Observations/Comments*
- The surprising part was the regionservers were trying to make so many
requests for the same block in the same minute (let alone hour). Could this
happen because the original request took a few seconds and so the
regionserver re-requested ? I didn't see any fetch errors in the
regionserver logs for blocks.
- Even more strange; my heap size was at 11G and the time when this was
happening, the used heap was at 2-4G. I would have expected the heap to
grow higher than that since the blockCache should be using atleast 40% of
the available heap space.
- Another strange thing that I observed was, the block was being requested
from the same datanode every single time.

*Possible Solution/Changes*
- Would it make sense to give remote blocks higher priority over the local
blocks that can be read via SCR and not let them get evicted if there is a
tie in which block to evict ?
- Should we throttle the number of outgoing requests for a block ? I am not
sure if my firewall caused some issue but I wouldn't expect multiple block
fetch requests in the same minute. I did see a few RST packets getting
dropped at the firewall but I wasn't able to trace the problem was due to
this.
- We have 3 replicas available, shouldn't we request from the other
datanode if one might take a lot of time ? The amount of time it took to
read a block went up when the box was under heavy load, yet the re-requests
were going to the same one. Is this something that is available on the
DFSClient and can we exploit it ?
- Is it possible to migrate a region to a server which has higher number of
blocks available for it ? We don't need to make this automatic, but we
could provide a command that could be invoked manually to assign a region
to a specific regionserver. Thoughts ?

Thanks,
Viral

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please imme

RE: GC recommendations for large Region Server heaps

[Vladimir Rodionov]

>>I'm especially concerned about long pauses causing zk session timeouts and
>>consequent RS shutdowns. Our boxes do have a lot of RAM and we are
>>exploring how we can use more of it for the cluster while maintaining
>>overall stability.

Suraj, what is the maximum GC pause you have observed in your cluster so far?
I do not think s-t-w GC pause on 8GB heap will lasts longer than 5-8 secs.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Suraj Varma [svarma...@gmail.com]
Sent: Monday, July 08, 2013 11:09 AM
To: user@hbase.apache.org
Subject: GC recommendations for large Region Server heaps

Hello:
We have an HBase cluster with region servers running on 8GB heap size with
a 0.6 block cache (it is a read heavy cluster, with bursty write traffic
via MR jobs). (version: hbase-0.94.6.1)

During HBaseCon, while speaking to a few attendees, I heard some folks were
running region servers as high as 24GB and some others in the 16GB range.

So - question: Are there any special GC recommendations (tuning parameters,
flags, etc) that folks who run at these large heaps can recommend while
moving up from an 8GB heap? i.e. for 16GB and for 24GB RS heaps ... ?

I'm especially concerned about long pauses causing zk session timeouts and
consequent RS shutdowns. Our boxes do have a lot of RAM and we are
exploring how we can use more of it for the cluster while maintaining
overall stability.

Also - if there are clusters running multiple region servers per host, I'd
be very interested to know what RS heap sizes those are being run at ...
and whether this was chosen as an alternative to running a single RS with
large heap.

(I know I'll have to test the GC stuff out on my cluster and for my
workloads anyway ... but just trying to get a feel of what sort of tuning
options had to be used to have a stable HBase cluster with 16 or 24GB RS
heaps).

Thanks in advance,
--Suraj

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: G1 before/after GC time graph

[Vladimir Rodionov]

What is  -Xmx heap size?

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Otis Gospodnetic [otis.gospodne...@gmail.com]
Sent: Wednesday, July 10, 2013 12:00 PM
To: user@hbase.apache.org
Subject: G1 before/after GC time graph

Hi,

There was talk of G1 and HBase here over the past few days.  I wanted
to share a fresh and telling SPM graph showing G1 + some other params
in action:

Result first: https://apps.sematext.com/spm/s/wN0hKVixxX

This is what we had before:
  -XX:+HeapDumpOnOutOfMemoryError -XX:+UseG1GC
-XX:InitiatingHeapOccupancyPercent=35 -XX:G1ReservePercent=25

And then we added:
  -XX:MaxGCPauseMillis=200 -XX:NewRatio=4

I hope this helps.

Otis
--
HBase Performance Monitoring -- http://sematext.com/spm

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: G1 before/after GC time graph

[Vladimir Rodionov]

+1

1.8G heap is on a very low side. 32-64GB is more interesting cases.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Kevin O'dell [kevin.od...@cloudera.com]
Sent: Wednesday, July 10, 2013 2:27 PM
To: user@hbase.apache.org
Subject: Re: G1 before/after GC time graph

Anyway you can test this with a 32GB heap? :)


On Wed, Jul 10, 2013 at 5:23 PM, Otis Gospodnetic <
otis.gospodne...@gmail.com> wrote:

> But, very importantly:
>
> CPU after changes: https://apps.sematext.com/spm/s/CpZ9aPhL9L
> Load after changes: https://apps.sematext.com/spm/s/i7f3x6MXdZ
>
> So in practical terms, use something like -XX:MaxGCPauseMillis=200
> -XX:NewRatio=4 to force short GC pauses, but count on having more of
> them and CPU sweating more.  This gives you smaller pauses, no big STW
> pauses, but be prepared to pay in CPU cycles.
>
> HTH
>
> Otis
> --
> HBase Performance Monitoring -- http://sematext.com/spm
>
>
>
> On Wed, Jul 10, 2013 at 4:20 PM, Otis Gospodnetic
>  wrote:
> > Not big in this case - 1800m
> >
> > Otis
> > --
> > HBase Performance Monitoring -- http://sematext.com/spm
> >
> >
> >
> > On Wed, Jul 10, 2013 at 3:57 PM, Vladimir Rodionov
> >  wrote:
> >>
> >> What is  -Xmx heap size?
> >>
> >> Best regards,
> >> Vladimir Rodionov
> >> Principal Platform Engineer
> >> Carrier IQ, www.carrieriq.com
> >> e-mail: vrodio...@carrieriq.com
> >>
> >> 
> >> From: Otis Gospodnetic [otis.gospodne...@gmail.com]
> >> Sent: Wednesday, July 10, 2013 12:00 PM
> >> To: user@hbase.apache.org
> >> Subject: G1 before/after GC time graph
> >>
> >> Hi,
> >>
> >> There was talk of G1 and HBase here over the past few days.  I wanted
> >> to share a fresh and telling SPM graph showing G1 + some other params
> >> in action:
> >>
> >> Result first: https://apps.sematext.com/spm/s/wN0hKVixxX
> >>
> >> This is what we had before:
> >>   -XX:+HeapDumpOnOutOfMemoryError -XX:+UseG1GC
> >> -XX:InitiatingHeapOccupancyPercent=35 -XX:G1ReservePercent=25
> >>
> >> And then we added:
> >>   -XX:MaxGCPauseMillis=200 -XX:NewRatio=4
> >>
> >> I hope this helps.
> >>
> >> Otis
> >> --
> >> HBase Performance Monitoring -- http://sematext.com/spm
> >>
> >> Confidentiality Notice:  The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited.  If you have received this message in error, please
> immediately notify the sender and/or notificati...@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>



--
Kevin O'Dell
Systems Engineer, Cloudera

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

Master server abort

[Vladimir Rodionov]

:41:33,863 WARN 
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node 
/hbase/table/TRIAL-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,876 WARN 
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node 
/hbase/table/TRIAL-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,889 WARN 
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node 
/hbase/table/packageindex already deleted, and this is not a retry
2013-07-10 19:41:56,349 WARN org.apache.hadoop.hbase.master.AssignmentManager: 
Received OPENED for region 61e36043936d733a370c8f6dddb83566 from server 
sjc1-eng-qa04.carrieriq.com,60020,1373485278882 but region was in  the state 
INDEX_TEST3-IDPROFILES,16,1363283705765.61e36043936d733a370c8f6dddb83566. 
state=OPEN, ts=1373485312867, 
server=sjc1-eng-qa04.carrieriq.com,60020,1373485278882 and not in expected 
OFFLINE, PENDING_OPEN or OPENING states

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Otis Gospodnetic [otis.gospodne...@gmail.com]
Sent: Thursday, July 11, 2013 10:55 AM
To: user@hbase.apache.org
Subject: Re: G1 before/after GC time graph

Correct!

Otis
Solr & ElasticSearch Support
http://sematext.com/
On Jul 11, 2013 1:18 PM, "Asaf Mesika"  wrote:

> This means you can safely run Hadoop and Hbase on jvm 7?
> We were just considering switching in production to java 7.
>
> On Thursday, July 11, 2013, Azuryy Yu wrote:
>
> > Otis,
> >
> > I will do this test, maybe on the end of this month. because I haven't
> big
> > memory server for test now util the end of this month.
> >
> >
> >
> >
> > On Thu, Jul 11, 2013 at 9:38 AM, Otis Gospodnetic <
> > otis.gospodne...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > Probably not any time in the near future.  We could do it, plop SPM in
> > > it and watch, but don't have time for it now.
> > >
> > > If anyone wants to give it a shot and use SPM to capture various
> > > JVM/HBase/server metrics, let me know if help around SPM is needed,
> > > longer trials, or whatever.
> > >
> > > Otis
> > > --
> > > HBase Performance Monitoring -- http://sematext.com/spm
> > >
> > >
> > >
> > > On Wed, Jul 10, 2013 at 5:27 PM, Kevin O'dell <
> kevin.od...@cloudera.com>
> > > wrote:
> > > > Anyway you can test this with a 32GB heap? :)
> > > >
> > > >
> > > > On Wed, Jul 10, 2013 at 5:23 PM, Otis Gospodnetic <
> > > > otis.gospodne...@gmail.com> wrote:
> > > >
> > > >> But, very importantly:
> > > >>
> > > >> CPU after changes: https://apps.sematext.com/spm/s/CpZ9aPhL9L
> > > >> Load after changes: https://apps.sematext.com/spm/s/i7f3x6MXdZ
> > > >>
> > > >> So in practical terms, use something like -XX:MaxGCPauseMillis=200
> > > >> -XX:NewRatio=4 to force short GC pauses, but count on having more of
> > > >> them and CPU sweating more.  This gives you smaller pauses, no big
> STW
> > > >> pauses, but be prepared to pay in CPU cycles.
> > > >>
> > > >> HTH
> > > >>
> > > >> Otis
> > > >> --
> > > >> HBase Performance Monitoring -- http://sematext.com/spm
> > > >>
> > > >>
> > > >>
> > > >> On Wed, Jul 10, 2013 at 4:20 PM, Otis Gospodnetic
> > > >>  wrote:
> > > >> > Not big in this case - 1800m
> > > >> >
> > > >> > Otis
> > > >> > --
> > > >> > HBase Performance Monitoring -- http://sematext.com/spm
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Wed, Jul 10, 2013 at 3:57 PM, Vladimir Rodionov
> > > >> >  wrote:
> > > >> >>
> > > >> >> What is  -Xmx heap size?
> > > >> >>
> > > >> >> Best regards,
> > > >> >> Vladimir Rodionov
> > > >> >> Principal Platform Engineer
> > > >> >> Carrier IQ, www.carrieriq.com
> > > >> >> e-mail: vrodio...@carrieriq.com
> > > >> >>
> > > >> >> 
> > > >> >> From: Otis Gospodnetic [otis.gospodne...@gmail.com]
> > > >> >> Sent: Wednesday, July 10, 2013 12:00 PM
> > > >> >> To: user@hb

RE: Programmatically create HTable with snappy-compressed columns

[Vladimir Rodionov]

Here it is (the simplest one):

HColumnDescriptor fam = new HColumnDescriptor(familyName);
fam.setCompressionType(Algorithm.SNAPPY);
HTableDescriptor table = new HTableDescriptor(tableName);
table.addFamily(fam);
HBaseAdmin admin = new HBaseAdmin(HBaseConfiguration.create());
admin.createTable(table);

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Flavio Pompermaier [pomperma...@okkam.it]
Sent: Tuesday, July 16, 2013 6:08 AM
To: user@hbase.apache.org
Subject: Programmatically create HTable with snappy-compressed columns

Hi to all,
I have to programmatically create HTable (from Java) with a compressed
(snappy) column.
Is it possible to do it from code or do I have to manually create them via
hbase shell?
If it is possible could someone show me a snippet?

Best,
Flavio

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

Re: [ANNOUNCE] Secondary Index in HBase - from Huawei

[Vladimir Rodionov]

Excellent job, congrats Huawei team. Phoenix + hindex integration?

-Vladimir


On Tue, Aug 13, 2013 at 12:53 AM, Subroto  wrote:

> Good to see this. Thanks a lot to Priyank, Ramakrishna, Rajesh.
>
> _
>/(|
>   (  :
>  __\  \  _
>()  `|
>   ()|   |
>().__|
> (___)__.|_
>
>
> Cheers,
> Subroto Sanyal
>
> On Aug 13, 2013, at 9:28 AM, ramkrishna vasudevan wrote:
>
> > Good to see this.  Hope this would help in more improvements and
> > enhancements. :)
> >
> >
> > On Tue, Aug 13, 2013 at 12:14 PM, Anoop John 
> wrote:
> >
> >> Good to see this Rajesh. Thanks a lot to Huawei HBase team!
> >>
> >> -Anoop-
> >>
> >> On Tue, Aug 13, 2013 at 11:49 AM, rajeshbabu chintaguntla <
> >> rajeshbabu.chintagun...@huawei.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> We have been working on implementing secondary index in HBase, and had
> >>> shared an overview of our design in the 2012  Hadoop Technical
> Conference
> >>> at Beijing(http://bit.ly/hbtc12-hindex). We are pleased to open source
> >> it
> >>> today.
> >>>
> >>> The project is available on github.
> >>>  https://github.com/Huawei-Hadoop/hindex
> >>>
> >>> It is 100% Java, compatible with Apache HBase 0.94.8, and is open
> sourced
> >>> under Apache Software License v2.
> >>>
> >>> Following features are supported currently.
> >>> -  multiple indexes on table,
> >>> -  multi column index,
> >>> -  index based on part of a column value,
> >>> -  equals and range condition scans using index, and
> >>> -  bulk loading data to indexed table (Indexing done with bulk
> >>> load)
> >>>
> >>> We now plan to raise HBase JIRA(s) to make it available in Apache
> >> release,
> >>> and can hopefully continue our work on this in the community.
> >>>
> >>> Regards
> >>> Rajeshbabu
> >>>
> >>>
> >>
>
>

RE: issue about rowkey design

[Vladimir Rodionov]

Secondary index requires multiple random seeks and is not efficient in many 
cases.
What you need is different row_keys (one for each request type)

user_id, session_id, visit_time =>

rowkey1 => "q1", visit_time, user_id
rowkey2 => "q2", visit_time, session_id
rowkey3 => "q3", user_id, session_id : ts = visit_time


Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: fga...@despegar.com [fga...@despegar.com]
Sent: Sunday, August 18, 2013 6:25 PM
To: user@hbase.apache.org; Kiru Pakkirisamy
Subject: Re: issue about rowkey design

You can use a secondary table as a 'secondary index' setting your row as value 
(or column) in it.
Enviado desde mi BlackBerry de Personal (http://www.personal.com.ar/)

-Original Message-
From: ch huang 
Date: Mon, 19 Aug 2013 09:05:19
To: ; Kiru Pakkirisamy
Reply-To: user@hbase.apache.org
Subject: Re: issue about rowkey design

what you mean secondary index? has hbase secondary index?

On Sat, Aug 17, 2013 at 12:48 AM, Kiru Pakkirisamy <
kirupakkiris...@yahoo.com> wrote:

> We did design with something equivalent to userid as the key and all the
> user sessions in there.
> But when we tried to look for particular user sessions within a time
> range, we found the ColumnPrefixFilter (say on the timerange) did not give
> us much performance.
> So we ended up creating another table with time-range as key and all the
> user sessions ids in it (equivalent).
> I am pretty much repeating Bryan, but if you just use the ids, you do not
> duplicate that much data (called secondary index ?)
>
> Regards,
> - kiru
>
>
> Kiru Pakkirisamy | webcloudtech.wordpress.com
>
>
> 
>  From: Bryan Beaudreault 
> To: user@hbase.apache.org
> Sent: Friday, August 16, 2013 8:06 AM
> Subject: Re: issue about rowkey design
>
>
> HBase is all about denormalization and designing for the usecase/query
> pattern.   If it's possible for your application it will be better to
> provide three different indexes, as opposed to fitting them all into one
> rowkey design.
>
>
> On Fri, Aug 16, 2013 at 5:33 AM, ch huang  wrote:
>
> > hi,all
> >  i have data (data  is very huge) with user id ,session id ,and visit
> > time. my query pattern is ,"find all user id in certain time range,find
> one
> > user's all session id ,and find all session id in certain time range".
> >my difficult is that i can not find a rowkey that good for all the
> > search pattern, i wonder if i need set three rowkey for these search
> > patterns,it's say i need triple my data storage ,any good idea?
> >
>


Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: I am curious how to find hbase replication lag

[Vladimir Rodionov]

Just simple canary probe approach

Update cluster1:t1:canary_row with a current time every (say) 1 sec
Read time from cluster2:t1:canary_row every second

Compute the difference


Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Demai Ni [nid...@gmail.com]
Sent: Monday, August 19, 2013 1:28 PM
To: user@hbase.apache.org
Subject: Re: I am curious how to find hbase replication lag

Ted, thanks for connecting the two discussion. a topic that quite some
folks are looking for solutions.

Alex, as far as my study goes, there is no direct/easy way to get the lag
info that is easily consumable by regular users(i.e. not a hbase expert).
>From user perspecitive, the lag can be either time or quantity. Although
current hbase replication metrics contains a lot of good information, third
party tool/monitor has to be applied (JMX, ganglia, etc.)  On 0.94 level,
the metrics info is at regionsever level, on 95.0 such info also at peer
cluster level. I don't think there is plan for a table-level yet.

However, the use case is valid. For example: I am replicating table t1 from
cluster M1 to cluster S1; how many seconds/minutes the lag is? Well, I
haven't found a good solution yet. :-)

Demai


On Mon, Aug 19, 2013 at 1:04 PM, Ted Yu  wrote:

> This is related: http://search-hadoop.com/m/SrEIT1jtzPF
>
> Cheers
>
>
> On Mon, Aug 19, 2013 at 12:50 PM, gordoslocos 
> wrote:
>
> > I believe hbase keeps info in zk that gives you the count of pending
> > operations to be replicated. Check into the rz zookeeper node in the
> hbase
> > replication documentation.
> >
> > http://hbase.apache.org/replication.html
> >
> >
> > On 19/08/2013, at 16:42, Alex Newman  wrote:
> >
> > > I have setup HBase replication. I want to know how out of date my
> > replicant
> > > cluster is. How does one monitor that?
> > >
> > > -Alex Newman
> >
>

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: passing a parameter to an observer coprocessor

[Vladimir Rodionov]

Add fake cf + column to your Put operation

Put put = new Put(row);

put.addFamily("COMMAND".getBytes(), "flag".getBytes(),"true".getBytes());

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Wei Tan [w...@us.ibm.com]
Sent: Thursday, August 22, 2013 9:36 AM
To: hbase-u...@hadoop.apache.org
Subject: passing a parameter to an observer coprocessor

Hi all,

   I want to add some dynamic behavior to my observer cp, say:

postPut(){

if(flag) {do function1()};
else {do function2()}

}

  Is there a way to dynamically change the value of flag? One Feasible
approaches is to change a value in table descriptor, but then I need to
restart the table. Is there a way to pass a parameter using an API, say,
from an endpoint, to an observer? Another way to ask this question is, is
there a shared variable between an observer, and an endpoint, in a common
table?

Thanks,
Wei

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: one column family but lots of tables

[Vladimir Rodionov]

Nope. Column family is per table (its sub-directory inside your table directory 
in HDFS).
If you have N tables you will always have , at least, N distinct CFs (even if 
they have the same name).

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Koert Kuipers [ko...@tresata.com]
Sent: Thursday, August 22, 2013 8:06 AM
To: user@hbase.apache.org
Subject: one column family but lots of tables

i read in multiple places that i should try to limit the number of column
families in hbase.

do i understand it correctly that when i create lots of tables, but they
all use the same column family (by name), that i am just using one column
family and i am OK with respect to limiting number of column families ?

thanks! koert

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: passing a parameter to an observer coprocessor

[Vladimir Rodionov]

Sorry, CF must not be fake, I suppose (because its region coprocessor)

put.add(REAL_COLUMN_FAMILY, "flag".getBytes(),"true".getBytes());

'flag' is the fake column. You have to process these columns in your 
Coprocessor and extract them.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com

____
From: Vladimir Rodionov
Sent: Thursday, August 22, 2013 10:53 AM
To: user@hbase.apache.org; hbase-u...@hadoop.apache.org
Subject: RE: passing a parameter to an observer coprocessor

Add fake cf + column to your Put operation

Put put = new Put(row);

put.addFamily("COMMAND".getBytes(), "flag".getBytes(),"true".getBytes());

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Wei Tan [w...@us.ibm.com]
Sent: Thursday, August 22, 2013 9:36 AM
To: hbase-u...@hadoop.apache.org
Subject: passing a parameter to an observer coprocessor

Hi all,

   I want to add some dynamic behavior to my observer cp, say:

postPut(){

if(flag) {do function1()};
else {do function2()}

}

  Is there a way to dynamically change the value of flag? One Feasible
approaches is to change a value in table descriptor, but then I need to
restart the table. Is there a way to pass a parameter using an API, say,
from an endpoint, to an observer? Another way to ask this question is, is
there a shared variable between an observer, and an endpoint, in a common
table?

Thanks,
Wei

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: one column family but lots of tables

[Vladimir Rodionov]

Yes, number of tables must be reasonable as well. Region Servers operates on 
'regions' .
Each Table can have multiple CFs, each CF can have multiple regions. The more 
regions you have per Region Server -
the more data you will need to keep in memory, the more time it will take to 
recover  from Region Server failure, therefore 
there is a practical limit on number of regions per one Region Server (hundreds 
- not thousands). It is recommended
to keep the number of regions per server much less than 1000 (say , 100- 500 
max). 


Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Koert Kuipers [ko...@tresata.com]
Sent: Thursday, August 22, 2013 12:30 PM
To: user@hbase.apache.org; Vladimir Rodionov
Subject: Re: one column family but lots of tables

if that is the case, how come people keep warning about limiting the number of 
column families to only a handful (with more hbase performance will degrade 
supposedly), yet there seems to be no similar warnings for number of tables? 
see for example here:
http://comments.gmane.org/gmane.comp.java.hadoop.hbase.user/27616

if a table means at least one column family then the number of tables should 
also be kept to a minumum, no?




On Thu, Aug 22, 2013 at 1:58 PM, Vladimir Rodionov 
mailto:vrodio...@carrieriq.com>> wrote:
Nope. Column family is per table (its sub-directory inside your table directory 
in HDFS).
If you have N tables you will always have , at least, N distinct CFs (even if 
they have the same name).

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com<http://www.carrieriq.com>
e-mail: vrodio...@carrieriq.com<mailto:vrodio...@carrieriq.com>


From: Koert Kuipers [ko...@tresata.com<mailto:ko...@tresata.com>]
Sent: Thursday, August 22, 2013 8:06 AM
To: user@hbase.apache.org<mailto:user@hbase.apache.org>
Subject: one column family but lots of tables

i read in multiple places that i should try to limit the number of column
families in hbase.

do i understand it correctly that when i create lots of tables, but they
all use the same column family (by name), that i am just using one column
family and i am OK with respect to limiting number of column families ?

thanks! koert

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or 
notificati...@carrieriq.com<mailto:notificati...@carrieriq.com> and delete or 
destroy any copy of this message and its attachments.

RE: passing a parameter to an observer coprocessor

[Vladimir Rodionov]

This is not so flexible and dynamic. Changing table level attributes
will require disabling/enabling table (20-30 seconds for large tables)
Is OK with your use case?

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Wei Tan [w...@us.ibm.com]
Sent: Thursday, August 22, 2013 1:32 PM
To: user@hbase.apache.org
Cc: hbase-u...@hadoop.apache.org
Subject: Re: passing a parameter to an observer coprocessor

Hi Anoop and Vladimir,

   Thanks for your reply. I think adding an attribute to each Mutation is
not the flexibility level we want -- also we do not want that level of
overhead. Having a singleton class acting as a global variable for a
table, using endpoint to set and letting observer to read, is more
desirable.

Wei


From:   Anoop John 
To: user@hbase.apache.org,
Cc: "hbase-u...@hadoop.apache.org" 
Date:   08/22/2013 03:03 PM
Subject:Re: passing a parameter to an observer coprocessor



This will need you have to pass the attr with every Mutation. If this
level
of dynamic nature you dont want, then as Andy said can impl Observer and
Endpoint and some Singleton object which both can share..

-Anoop-

On Fri, Aug 23, 2013 at 12:18 AM, Anoop John 
wrote:

> Can use Mutation#setAttribute(String name, byte[] value)  ?Based on
> this attr value can decide in CP which flow it should go with?
>
> -Anoop-
>
>  On Thu, Aug 22, 2013 at 11:33 PM, Vladimir Rodionov <
> vrodio...@carrieriq.com> wrote:
>
>> Sorry, CF must not be fake, I suppose (because its region coprocessor)
>>
>> put.add(REAL_COLUMN_FAMILY, "flag".getBytes(),"true".getBytes());
>>
>> 'flag' is the fake column. You have to process these columns in your
>> Coprocessor and extract them.
>>
>> Best regards,
>> Vladimir Rodionov
>> Principal Platform Engineer
>> Carrier IQ, www.carrieriq.com
>> e-mail: vrodio...@carrieriq.com
>>
>> 
>> From: Vladimir Rodionov
>> Sent: Thursday, August 22, 2013 10:53 AM
>> To: user@hbase.apache.org; hbase-u...@hadoop.apache.org
>> Subject: RE: passing a parameter to an observer coprocessor
>>
>> Add fake cf + column to your Put operation
>>
>> Put put = new Put(row);
>>
>> put.addFamily("COMMAND".getBytes(),
"flag".getBytes(),"true".getBytes());
>>
>> Best regards,
>> Vladimir Rodionov
>> Principal Platform Engineer
>> Carrier IQ, www.carrieriq.com
>> e-mail: vrodio...@carrieriq.com
>>
>> 
>> From: Wei Tan [w...@us.ibm.com]
>> Sent: Thursday, August 22, 2013 9:36 AM
>> To: hbase-u...@hadoop.apache.org
>> Subject: passing a parameter to an observer coprocessor
>>
>> Hi all,
>>
>>I want to add some dynamic behavior to my observer cp, say:
>>
>> postPut(){
>>
>> if(flag) {do function1()};
>> else {do function2()}
>>
>> }
>>
>>   Is there a way to dynamically change the value of flag? One Feasible
>> approaches is to change a value in table descriptor, but then I need to
>> restart the table. Is there a way to pass a parameter using an API,
say,
>> from an endpoint, to an observer? Another way to ask this question is,
is
>> there a shared variable between an observer, and an endpoint, in a
common
>> table?
>>
>> Thanks,
>> Wei
>>
>> Confidentiality Notice:  The information contained in this message,
>> including any attachments hereto, may be confidential and is intended
to be
>> read only by the individual or entity to whom this message is
addressed. If
>> the reader of this message is not the intended recipient or an agent or
>> designee of the intended recipient, please note that any review, use,
>> disclosure or distribution of this message or its attachments, in any
form,
>> is strictly prohibited.  If you have received this message in error,
please
>> immediately notify the sender and/or notificati...@carrieriq.com and
>> delete or destroy any copy of this message and its attachments.
>>
>
>


Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: experiencing high latency for few reads in HBase

[Vladimir Rodionov]

1. 4 sec max latency is not that bad taking into account 12GB heap.  It can be 
much larger. What is your SLA?
2. Block evictions is the result of a poor cache hit rate and the root cause of 
a periodic stop-the-world GC pauses (max latencies
latencies you have been observing in the test)
3. Block cache consists of 3 parts (25% young generation, 50% - tenured, 25% - 
permanent). Permanent part is for CF with
IN_MEMORY = true (you can specify this when you create CF).  Block first stored 
in 'young gen' space, then gets promoted to 'tenured gen' space
(or gets evicted). May be your 'perm gen' space is underutilized? This is exact 
25% of 4GB (1GB). Although HBase LruBlockCache should use all the space 
allocated for block cache -
there is no guarantee (as usual). If you don have in_memory column families you 
may decrease



Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Saurabh Yahoo [saurabh...@yahoo.com]
Sent: Wednesday, August 28, 2013 5:10 AM
To: user@hbase.apache.org
Subject: experiencing high latency for few reads in HBase

Hi,

We are running a stress test in our 5 node cluster and we are getting the 
expected mean latency of 10ms. But we are seeing around 20 reads out of 25 
million reads having latency more than 4 seconds. Can anyone provide the 
insight what we can do to meet below second SLA for each and every read?

We observe the following things -

1. Reads are evenly distributed among 5 nodes.  CPUs remain under 5% utilized.

2. We have 4gb block cache (30% block cache out of 12gb) setup. 3gb block cache 
got filled up but around 1gb remained free. There are a large number of cache 
eviction.

Questions to experts -

1. If there are still 1gb of free block cache available, why is hbase evicting 
the block from cache?

4. We are seeing memory went up to 10gb three times before dropping sharply to 
5gb.

Any help is highly appreciable,

Thanks,
Saurabh.

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: experiencing high latency for few reads in HBase

[Vladimir Rodionov]

Just ignore last part: 'If you don have in_memory column families you may 
decrease'

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com

____
From: Vladimir Rodionov
Sent: Wednesday, August 28, 2013 12:15 PM
To: user@hbase.apache.org
Subject: RE: experiencing high latency for few reads in HBase

1. 4 sec max latency is not that bad taking into account 12GB heap.  It can be 
much larger. What is your SLA?
2. Block evictions is the result of a poor cache hit rate and the root cause of 
a periodic stop-the-world GC pauses (max latencies
latencies you have been observing in the test)
3. Block cache consists of 3 parts (25% young generation, 50% - tenured, 25% - 
permanent). Permanent part is for CF with
IN_MEMORY = true (you can specify this when you create CF).  Block first stored 
in 'young gen' space, then gets promoted to 'tenured gen' space
(or gets evicted). May be your 'perm gen' space is underutilized? This is exact 
25% of 4GB (1GB). Although HBase LruBlockCache should use all the space 
allocated for block cache -
there is no guarantee (as usual). If you don have in_memory column families you 
may decrease



Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Saurabh Yahoo [saurabh...@yahoo.com]
Sent: Wednesday, August 28, 2013 5:10 AM
To: user@hbase.apache.org
Subject: experiencing high latency for few reads in HBase

Hi,

We are running a stress test in our 5 node cluster and we are getting the 
expected mean latency of 10ms. But we are seeing around 20 reads out of 25 
million reads having latency more than 4 seconds. Can anyone provide the 
insight what we can do to meet below second SLA for each and every read?

We observe the following things -

1. Reads are evenly distributed among 5 nodes.  CPUs remain under 5% utilized.

2. We have 4gb block cache (30% block cache out of 12gb) setup. 3gb block cache 
got filled up but around 1gb remained free. There are a large number of cache 
eviction.

Questions to experts -

1. If there are still 1gb of free block cache available, why is hbase evicting 
the block from cache?

4. We are seeing memory went up to 10gb three times before dropping sharply to 
5gb.

Any help is highly appreciable,

Thanks,
Saurabh.

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: experiencing high latency for few reads in HBase

[Vladimir Rodionov]

Increasing Java heap size will make latency worse, actually.
You can't guarantee 1 sec max latency if run Java app (unless your heap size is 
much less than 1GB). 
I have never heard about strict maximum latency limit. Usually , its 99% , 99.9 
or 99.99% query percentiles.

You can greatly reduce your 99.xxx% percentile latency by storing you data in 2 
replicas to two different region servers.
Issue two read operations to those two region servers in parallel and get the 
first response. Probability theory states that  probability 
of two independent events (slow requests) is  the product of event's 
probabilities themselves. 


Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Saurabh Yahoo [saurabh...@yahoo.com]
Sent: Wednesday, August 28, 2013 4:18 PM
To: user@hbase.apache.org
Subject: Re: experiencing high latency for few reads in HBase

Thanks Kiru,

Scan is not an option for our use cases.  Our read is pretty random.

Any other suggestion to bring down the latency.

Thanks,
Saurabh.


On Aug 28, 2013, at 7:01 PM, Kiru Pakkirisamy  wrote:

> Saurabh, we are able to 600K rowxcolumns in 400 msec. We have put what was a 
> 40million row table as 400K rows and columns. We Get about 100 of the rows 
> from this 400K , do quite a bit of calculations in the coprocessor (almost a 
> group-order by) and return in this time.
> Maybe should consider replacing the MultiGets with Scan with Filter. I like 
> the FuzzyRowFilter even though you might need to match with exact key. It 
> works only with fixed length key.
> (I do have an issue right now, it is not scaling to multiple clients.)
>
> Regards,
> - kiru
>
>
> Kiru Pakkirisamy | webcloudtech.wordpress.com
>
>
> 
> From: Saurabh Yahoo 
> To: "user@hbase.apache.org" 
> Cc: "user@hbase.apache.org" 
> Sent: Wednesday, August 28, 2013 3:20 PM
> Subject: Re: experiencing high latency for few reads in HBase
>
>
> Thanks Kitu. We need less than 1 sec latency.
>
> We are using both muliGet and get.
>
> We have three concurrent clients running 10 threads each. ( that makes total 
> 30 concurrent clients).
>
> Thanks,
> Saurabh.
>
> On Aug 28, 2013, at 4:30 PM, Kiru Pakkirisamy  
> wrote:
>
>> Right 4 sec is good.
>> @Saurabh - so your read is - getting 20 out of 25 millions rows ?. Is this a 
>> Get or a Scan ?
>> BTW, in this stress test how many concurrent clients do you have ?
>>
>> Regards,
>> - kiru
>>
>>
>> 
>> From: Vladimir Rodionov 
>> To: "user@hbase.apache.org" 
>> Sent: Wednesday, August 28, 2013 12:15 PM
>> Subject: RE: experiencing high latency for few reads in HBase
>>
>>
>> 1. 4 sec max latency is not that bad taking into account 12GB heap.  It can 
>> be much larger. What is your SLA?
>> 2. Block evictions is the result of a poor cache hit rate and the root cause 
>> of a periodic stop-the-world GC pauses (max latencies
>>  latencies you have been observing in the test)
>> 3. Block cache consists of 3 parts (25% young generation, 50% - tenured, 25% 
>> - permanent). Permanent part is for CF with
>> IN_MEMORY = true (you can specify this when you create CF).  Block first 
>> stored in 'young gen' space, then gets promoted to 'tenured gen' space
>> (or gets evicted). May be your 'perm gen' space is underutilized? This is 
>> exact 25% of 4GB (1GB). Although HBase LruBlockCache should use all the 
>> space allocated for block cache -
>> there is no guarantee (as usual). If you don have in_memory column families 
>> you may decrease
>>
>>
>>
>> Best regards,
>> Vladimir Rodionov
>> Principal Platform Engineer
>> Carrier IQ, www.carrieriq.com
>> e-mail: vrodio...@carrieriq.com
>>
>> 
>> From: Saurabh Yahoo [saurabh...@yahoo.com]
>> Sent: Wednesday, August 28, 2013 5:10 AM
>> To: user@hbase.apache.org
>> Subject: experiencing high latency for few reads in HBase
>>
>> Hi,
>>
>> We are running a stress test in our 5 node cluster and we are getting the 
>> expected mean latency of 10ms. But we are seeing around 20 reads out of 25 
>> million reads having latency more than 4 seconds. Can anyone provide the 
>> insight what we can do to meet below second SLA for each and every read?
>>
>> We observe the following things -
>>
>> 1. Reads are evenly distributed among 5 nodes.  CPUs remain under 5% 
>> utilized.
>>
>> 2. W

RE: experiencing high latency for few reads in HBase

[Vladimir Rodionov]

Yes. HBase won't guarantee strict sub-second latency. 

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Saurabh Yahoo [saurabh...@yahoo.com]
Sent: Thursday, August 29, 2013 2:49 AM
To: user@hbase.apache.org
Cc: user@hbase.apache.org
Subject: Re: experiencing high latency for few reads in HBase

Hi Vlad,

We do have strict latency requirement as it is financial data requiring direct 
access from clients.

Are you saying that it is not possible to achieve sub second latency using 
hbase (because it is based on java.) ?







On Aug 28, 2013, at 8:10 PM, Vladimir Rodionov  wrote:

> Increasing Java heap size will make latency worse, actually.
> You can't guarantee 1 sec max latency if run Java app (unless your heap size 
> is much less than 1GB).
> I have never heard about strict maximum latency limit. Usually , its 99% , 
> 99.9 or 99.99% query percentiles.
>
> You can greatly reduce your 99.xxx% percentile latency by storing you data in 
> 2 replicas to two different region servers.
> Issue two read operations to those two region servers in parallel and get the 
> first response. Probability theory states that  probability
> of two independent events (slow requests) is  the product of event's 
> probabilities themselves.
>
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodio...@carrieriq.com
>
> 
> From: Saurabh Yahoo [saurabh...@yahoo.com]
> Sent: Wednesday, August 28, 2013 4:18 PM
> To: user@hbase.apache.org
> Subject: Re: experiencing high latency for few reads in HBase
>
> Thanks Kiru,
>
> Scan is not an option for our use cases.  Our read is pretty random.
>
> Any other suggestion to bring down the latency.
>
> Thanks,
> Saurabh.
>
>
> On Aug 28, 2013, at 7:01 PM, Kiru Pakkirisamy  
> wrote:
>
>> Saurabh, we are able to 600K rowxcolumns in 400 msec. We have put what was a 
>> 40million row table as 400K rows and columns. We Get about 100 of the rows 
>> from this 400K , do quite a bit of calculations in the coprocessor (almost a 
>> group-order by) and return in this time.
>> Maybe should consider replacing the MultiGets with Scan with Filter. I like 
>> the FuzzyRowFilter even though you might need to match with exact key. It 
>> works only with fixed length key.
>> (I do have an issue right now, it is not scaling to multiple clients.)
>>
>> Regards,
>> - kiru
>>
>>
>> Kiru Pakkirisamy | webcloudtech.wordpress.com
>>
>>
>> 
>> From: Saurabh Yahoo 
>> To: "user@hbase.apache.org" 
>> Cc: "user@hbase.apache.org" 
>> Sent: Wednesday, August 28, 2013 3:20 PM
>> Subject: Re: experiencing high latency for few reads in HBase
>>
>>
>> Thanks Kitu. We need less than 1 sec latency.
>>
>> We are using both muliGet and get.
>>
>> We have three concurrent clients running 10 threads each. ( that makes total 
>> 30 concurrent clients).
>>
>> Thanks,
>> Saurabh.
>>
>> On Aug 28, 2013, at 4:30 PM, Kiru Pakkirisamy  
>> wrote:
>>
>>> Right 4 sec is good.
>>> @Saurabh - so your read is - getting 20 out of 25 millions rows ?. Is this 
>>> a Get or a Scan ?
>>> BTW, in this stress test how many concurrent clients do you have ?
>>>
>>> Regards,
>>> - kiru
>>>
>>>
>>> 
>>> From: Vladimir Rodionov 
>>> To: "user@hbase.apache.org" 
>>> Sent: Wednesday, August 28, 2013 12:15 PM
>>> Subject: RE: experiencing high latency for few reads in HBase
>>>
>>>
>>> 1. 4 sec max latency is not that bad taking into account 12GB heap.  It can 
>>> be much larger. What is your SLA?
>>> 2. Block evictions is the result of a poor cache hit rate and the root 
>>> cause of a periodic stop-the-world GC pauses (max latencies
>>> latencies you have been observing in the test)
>>> 3. Block cache consists of 3 parts (25% young generation, 50% - tenured, 
>>> 25% - permanent). Permanent part is for CF with
>>> IN_MEMORY = true (you can specify this when you create CF).  Block first 
>>> stored in 'young gen' space, then gets promoted to 'tenured gen' space
>>> (or gets evicted). May be your 'perm gen' space is underutilized? This is 
>>> exact 25% of 4GB (1GB). Although HBase LruBlockCache should use all the 
>&g

RE: experiencing high latency for few reads in HBase

[Vladimir Rodionov]

Usually, either cluster restart or major compaction helps improving locality 
index.
There is an issue in region assignment after table disable/enable in 0.94.x (x 
<11) which 
breaks HDFS locality. Fixed in 0.94.11 

You can write your own routine to manually "localize" particular table using 
public HBase Client API.

But this won't help you to stay withing 1 sec anyway. 
 
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Saurabh Yahoo [saurabh...@yahoo.com]
Sent: Thursday, August 29, 2013 10:52 AM
To: user@hbase.apache.org
Cc: user@hbase.apache.org
Subject: Re: experiencing high latency for few reads in HBase

Thanks Vlad.

Quick question. I notice hdfsBlocksLocalityIndex is around 50 in all region 
servers.

Does that could be a problem? If it is, how to solve that? We already ran the 
major compaction after ingesting the data.

Thanks,
Saurabh.

On Aug 29, 2013, at 12:17 PM, Vladimir Rodionov  wrote:

> Yes. HBase won't guarantee strict sub-second latency.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodio...@carrieriq.com
>
> 
> From: Saurabh Yahoo [saurabh...@yahoo.com]
> Sent: Thursday, August 29, 2013 2:49 AM
> To: user@hbase.apache.org
> Cc: user@hbase.apache.org
> Subject: Re: experiencing high latency for few reads in HBase
>
> Hi Vlad,
>
> We do have strict latency requirement as it is financial data requiring 
> direct access from clients.
>
> Are you saying that it is not possible to achieve sub second latency using 
> hbase (because it is based on java.) ?
>
>
>
>
>
>
>
> On Aug 28, 2013, at 8:10 PM, Vladimir Rodionov  
> wrote:
>
>> Increasing Java heap size will make latency worse, actually.
>> You can't guarantee 1 sec max latency if run Java app (unless your heap size 
>> is much less than 1GB).
>> I have never heard about strict maximum latency limit. Usually , its 99% , 
>> 99.9 or 99.99% query percentiles.
>>
>> You can greatly reduce your 99.xxx% percentile latency by storing you data 
>> in 2 replicas to two different region servers.
>> Issue two read operations to those two region servers in parallel and get 
>> the first response. Probability theory states that  probability
>> of two independent events (slow requests) is  the product of event's 
>> probabilities themselves.
>>
>>
>> Best regards,
>> Vladimir Rodionov
>> Principal Platform Engineer
>> Carrier IQ, www.carrieriq.com
>> e-mail: vrodio...@carrieriq.com
>>
>> 
>> From: Saurabh Yahoo [saurabh...@yahoo.com]
>> Sent: Wednesday, August 28, 2013 4:18 PM
>> To: user@hbase.apache.org
>> Subject: Re: experiencing high latency for few reads in HBase
>>
>> Thanks Kiru,
>>
>> Scan is not an option for our use cases.  Our read is pretty random.
>>
>> Any other suggestion to bring down the latency.
>>
>> Thanks,
>> Saurabh.
>>
>>
>> On Aug 28, 2013, at 7:01 PM, Kiru Pakkirisamy  
>> wrote:
>>
>>> Saurabh, we are able to 600K rowxcolumns in 400 msec. We have put what was 
>>> a 40million row table as 400K rows and columns. We Get about 100 of the 
>>> rows from this 400K , do quite a bit of calculations in the coprocessor 
>>> (almost a group-order by) and return in this time.
>>> Maybe should consider replacing the MultiGets with Scan with Filter. I like 
>>> the FuzzyRowFilter even though you might need to match with exact key. It 
>>> works only with fixed length key.
>>> (I do have an issue right now, it is not scaling to multiple clients.)
>>>
>>> Regards,
>>> - kiru
>>>
>>>
>>> Kiru Pakkirisamy | webcloudtech.wordpress.com
>>>
>>>
>>> 
>>> From: Saurabh Yahoo 
>>> To: "user@hbase.apache.org" 
>>> Cc: "user@hbase.apache.org" 
>>> Sent: Wednesday, August 28, 2013 3:20 PM
>>> Subject: Re: experiencing high latency for few reads in HBase
>>>
>>>
>>> Thanks Kitu. We need less than 1 sec latency.
>>>
>>> We are using both muliGet and get.
>>>
>>> We have three concurrent clients running 10 threads each. ( that makes 
>>> total 30 concurrent clients).
>>>
>>> Thanks,
>>> Saurabh.
>>>
>>> On Aug 28, 2013, at 4:30 PM, Kiru Pakkirisamy  
>>> wrot

RE: Coprocessor concurrency

[Vladimir Rodionov]

RS handler count 800 is an overkill. 

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Vladimir Rodionov
Sent: Friday, August 30, 2013 12:34 PM
To: user@hbase.apache.org; Kiru Pakkirisamy
Subject: RE: Coprocessor concurrency

Can you check RS thread stack traces during your test? If there are some 
concurrency/contention issues they will pop up in a stack traces.
Also, make sure that you run your client apps in a separate processes.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Kiru Pakkirisamy [kirupakkiris...@yahoo.com]
Sent: Friday, August 30, 2013 1:49 AM
To: hbase mailing list
Subject: Coprocessor concurrency

See below a null endpoint which takes 60-70msec on my 4 node ec2 cluster for a 
table with 45 regions.
When I run 64 concurrent clients for this the latency jumps to 3000-3700 msec.

(zookeeper maxClientCnxs is set to 0 (unlimited) and hbase regionserver handler 
count is 800). I hope I am not missing any configs for concurrency)

--

package com.serendio.hbase.coprocessor;

import java.io.IOException;
import java.util.Map;

import org.apache.hadoop.hbase.coprocessor.BaseEndpointCoprocessor;

//Aggregation implementation at a region.
public class SearchEndpoint extends BaseEndpointCoprocessor implements
SearchProtocol {
@Override
public Map foo(Map input, int topN)
throws IOException {
// Implement your logic here
// Map ret = new HashMap();
// ret.put("foo", 1.0);
return null;
}
}
===

package com.serendio.hbase.coprocessor;

import java.io.IOException;
import java.util.Map;

import org.apache.hadoop.hbase.ipc.CoprocessorProtocol;

public interface SearchProtocol extends CoprocessorProtocol {

public Map foo(Map input,
int topN) throws IOException;
}

==
package com.serendio.hbase.coprocessor;

import java.io.IOException;
import java.util.HashMap;
import java.util.Map;

import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.coprocessor.Batch.Call;

public class Search {
public static final String TEST_TABLE = "c4";
public static final String TEST_FAMILY = "info";
public static final String TEST_QUALIFIER = "c1";

/**
 * ``
 *
 * @param args
 */
public static void main(String[] args) {

org.apache.hadoop.conf.Configuration conf = HBaseConfiguration.create();
HTableInterface table = null;
try {
table = new HTable(conf, TEST_TABLE);

} catch (IOException e2) {
// TODO Auto-generated catch block
e2.printStackTrace();
}

Map> results = null;
final Map input = new HashMap();
input.put("test", 1.0);
final int topN = 10;
long start = System.currentTimeMillis();
try {
results = table.coprocessorExec(SearchProtocol.class, null, null,
new Call>() {

@Override
public Map call(SearchProtocol instance)
throws IOException {
// TODO Auto-generated method stub
return instance.foo(input, topN);
}
});
} catch (IOException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
} catch (Throwable e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
long end = System.currentTimeMillis();
System.out.println("Total search elapsed time: "
+ Long.toString(end - start));
}

}



Regards,
- kiru

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: Coprocessor concurrency

[Vladimir Rodionov]

Can you check RS thread stack traces during your test? If there are some 
concurrency/contention issues they will pop up in a stack traces.
Also, make sure that you run your client apps in a separate processes.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Kiru Pakkirisamy [kirupakkiris...@yahoo.com]
Sent: Friday, August 30, 2013 1:49 AM
To: hbase mailing list
Subject: Coprocessor concurrency

See below a null endpoint which takes 60-70msec on my 4 node ec2 cluster for a 
table with 45 regions.
When I run 64 concurrent clients for this the latency jumps to 3000-3700 msec.

(zookeeper maxClientCnxs is set to 0 (unlimited) and hbase regionserver handler 
count is 800). I hope I am not missing any configs for concurrency)

--

package com.serendio.hbase.coprocessor;

import java.io.IOException;
import java.util.Map;

import org.apache.hadoop.hbase.coprocessor.BaseEndpointCoprocessor;

//Aggregation implementation at a region.
public class SearchEndpoint extends BaseEndpointCoprocessor implements
SearchProtocol {
@Override
public Map foo(Map input, int topN)
throws IOException {
// Implement your logic here
// Map ret = new HashMap();
// ret.put("foo", 1.0);
return null;
}
}
===

package com.serendio.hbase.coprocessor;

import java.io.IOException;
import java.util.Map;

import org.apache.hadoop.hbase.ipc.CoprocessorProtocol;

public interface SearchProtocol extends CoprocessorProtocol {

public Map foo(Map input,
int topN) throws IOException;
}

==
package com.serendio.hbase.coprocessor;

import java.io.IOException;
import java.util.HashMap;
import java.util.Map;

import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.coprocessor.Batch.Call;

public class Search {
public static final String TEST_TABLE = "c4";
public static final String TEST_FAMILY = "info";
public static final String TEST_QUALIFIER = "c1";

/**
 * ``
 *
 * @param args
 */
public static void main(String[] args) {

org.apache.hadoop.conf.Configuration conf = HBaseConfiguration.create();
HTableInterface table = null;
try {
table = new HTable(conf, TEST_TABLE);

} catch (IOException e2) {
// TODO Auto-generated catch block
e2.printStackTrace();
}

Map> results = null;
final Map input = new HashMap();
input.put("test", 1.0);
final int topN = 10;
long start = System.currentTimeMillis();
try {
results = table.coprocessorExec(SearchProtocol.class, null, null,
new Call>() {

@Override
public Map call(SearchProtocol instance)
throws IOException {
// TODO Auto-generated method stub
return instance.foo(input, topN);
}
});
} catch (IOException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
} catch (Throwable e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
long end = System.currentTimeMillis();
System.out.println("Total search elapsed time: "
+ Long.toString(end - start));
}

}



Regards,
- kiru

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: Cannot start HBase master: loaded coprocessors are: []

[Vladimir Rodionov]

It has nothing to do with coprocessors. Your errors is Unresolved address.
Check "hbase.master.info.bindAddress" in hbase-site.xml

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: jaksky [stransky...@gmail.com]
Sent: Friday, September 06, 2013 4:54 AM
To: user@hbase.apache.org
Subject: Cannot start HBase master: loaded coprocessors are: []

In my HBase installation (distributed mode master + 2 region servers) with
external zookeeper running on top of Hadoop distributed installation (master
node + secondary master node + two data nodes) I am not able to start HBase
master. Region servers starts with no problem. HbaseMaster is failing with
error message:

2013-09-06 13:00:13,503 FATAL org.apache.hadoop.hbase.master.HMaster: Master
server abort: loaded coprocessors are: []
2013-09-06 13:00:13,503 FATAL org.apache.hadoop.hbase.master.HMaster:
Unhandled exception. Starting shutdown.
java.net.SocketException: Unresolved address
at sun.nio.ch.Net.translateToSocketException(Net.java:157)
at sun.nio.ch.Net.translateException(Net.java:183)
at sun.nio.ch.Net.translateException(Net.java:189)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:76)
at
org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
at org.apache.hadoop.http.HttpServer.start(HttpServer.java:602)
at
org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:913)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:557)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:391)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:208)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
... 6 more



--
View this message in context: 
http://apache-hbase.679495.n3.nabble.com/Cannot-start-HBase-master-loaded-coprocessors-are-tp4050328.html
Sent from the HBase User mailing list archive at Nabble.com.

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: Hbase ExportSnapshot with AWS

[Vladimir Rodionov]

There is no option to export materialized snapshot to AWS yet - only to another 
HBase cluster.
You can export snapshot metadata to s3 using S3 FileSystem  in Hadoop, but this 
is not you are looking for.
You can export snapshots to table(s) in another cluster and then use distcp to 
copy data over to s3.
One caveat: you have to have spare cluster for that, because you can't export 
table's snapshot to different cluster and
different table (name).

yea, too many limitations. One major enhancement to snapshots which is 
definitely welcomed is ability to export materialized
snapshots to a different DFS location (S3 in this case)


Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Dhanasekaran Anbalagan [bugcy...@gmail.com]
Sent: Monday, September 09, 2013 12:51 PM
To: cdh-user; user@hbase.apache.org
Subject: Hbase ExportSnapshot with AWS

Hi Guys,

It's possible to Export Snapshot with Amazon S3 or Glacier. Currently we
have two hbase cluster running on same datacenter.  We want
backup solution for Hbase data.

want redundant backup solution for data store. something like S3 or Glacier.

I will use CopyTable and snapshot for Hbase data. But the what is use
cluster are in same datacenter.

If two cluster are down. I want retrieve data from

Please guide me. what are the possible solution.

-Dhanasekaran.

Did I learn something today? If not, I wasted it.

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: Hbase ExportSnapshot with AWS

[Vladimir Rodionov]

Great, thanks.  One possible enhancement: support for  arbitrary file system 
(not only s3).

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Carlos Espinoza [car...@imgur.com]
Sent: Monday, September 09, 2013 2:57 PM
To: user@hbase.apache.org
Subject: Re: Hbase ExportSnapshot with AWS

You might be interested in a tool I recently wrote. It's a wrapper around
the Export Snapshot tool. It's not heavily tested, but it has worked for us
so far. You can use it to create a snapshot and export it to S3. It can
also be used to import a snapshot from S3 into a running HBase cluster. You
need to dedicate a bucket for this.

https://github.com/lospro7/snapshot-s3-util


On Mon, Sep 9, 2013 at 1:13 PM, Vladimir Rodionov
wrote:

> There is no option to export materialized snapshot to AWS yet - only to
> another HBase cluster.
> You can export snapshot metadata to s3 using S3 FileSystem  in Hadoop, but
> this is not you are looking for.
> You can export snapshots to table(s) in another cluster and then use
> distcp to copy data over to s3.
> One caveat: you have to have spare cluster for that, because you can't
> export table's snapshot to different cluster and
> different table (name).
>
> yea, too many limitations. One major enhancement to snapshots which is
> definitely welcomed is ability to export materialized
> snapshots to a different DFS location (S3 in this case)
>
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodio...@carrieriq.com
>
> 
> From: Dhanasekaran Anbalagan [bugcy...@gmail.com]
> Sent: Monday, September 09, 2013 12:51 PM
> To: cdh-user; user@hbase.apache.org
> Subject: Hbase ExportSnapshot with AWS
>
> Hi Guys,
>
> It's possible to Export Snapshot with Amazon S3 or Glacier. Currently we
> have two hbase cluster running on same datacenter.  We want
> backup solution for Hbase data.
>
> want redundant backup solution for data store. something like S3 or
> Glacier.
>
> I will use CopyTable and snapshot for Hbase data. But the what is use
> cluster are in same datacenter.
>
> If two cluster are down. I want retrieve data from
>
> Please guide me. what are the possible solution.
>
> -Dhanasekaran.
>
> Did I learn something today? If not, I wasted it.
>
> Confidentiality Notice:  The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited.  If you have received this message in error, please
> immediately notify the sender and/or notificati...@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: Tables gets Major Compacted even if they haven't changed

[Vladimir Rodionov]

HBase can run major compaction (even if table has not been updated) to purge 
expired data (TTL).

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Premal Shah [premal.j.s...@gmail.com]
Sent: Monday, September 09, 2013 9:02 PM
To: user
Subject: Tables gets Major Compacted even if they haven't changed

Hi,
We have a bunch on tables in our HBase cluster. We have a script which
makes sure all of them get Major Compacted once every 2 days. There are 2
things I'm observing

1) Table X has not updated in a month. We have not inserted, updated or
deleted data. However, it still major compacts every 2 days. All the
regions in this table have only 1 store file.

2) Table Y has a few regions where the rowkey is essentially a timestamp.
So, we only write to 1 region at a time. Over time, the region splits, and
then we write the one of the split regions. Now, whenever we major compact
the table, all regions get major compacted. Only 1 region has more than 1
store file, every other region has exactly once.

Is there a way to avoid compaction of regions that have not changed?

We are using HBase 0.94.11

--
Regards,
Premal Shah.

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: Tables gets Major Compacted even if they haven't changed

[Vladimir Rodionov]

Sure, you can overwrite standard compaction in RegionCoprocessor

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Premal Shah [premal.j.s...@gmail.com]
Sent: Monday, September 09, 2013 9:28 PM
To: user
Subject: Re: Tables gets Major Compacted even if they haven't changed

Ah ok. We don't expire any data, so have not set any TTLs. Is there a
policy we can use to avoid compacting regions that have not changed (ie
have just 1 store file)?


On Mon, Sep 9, 2013 at 9:13 PM, Vladimir Rodionov
wrote:

> HBase can run major compaction (even if table has not been updated) to
> purge expired data (TTL).
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodio...@carrieriq.com
>
> 
> From: Premal Shah [premal.j.s...@gmail.com]
> Sent: Monday, September 09, 2013 9:02 PM
> To: user
> Subject: Tables gets Major Compacted even if they haven't changed
>
> Hi,
> We have a bunch on tables in our HBase cluster. We have a script which
> makes sure all of them get Major Compacted once every 2 days. There are 2
> things I'm observing
>
> 1) Table X has not updated in a month. We have not inserted, updated or
> deleted data. However, it still major compacts every 2 days. All the
> regions in this table have only 1 store file.
>
> 2) Table Y has a few regions where the rowkey is essentially a timestamp.
> So, we only write to 1 region at a time. Over time, the region splits, and
> then we write the one of the split regions. Now, whenever we major compact
> the table, all regions get major compacted. Only 1 region has more than 1
> store file, every other region has exactly once.
>
> Is there a way to avoid compaction of regions that have not changed?
>
> We are using HBase 0.94.11
>
> --
> Regards,
> Premal Shah.
>
> Confidentiality Notice:  The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited.  If you have received this message in error, please
> immediately notify the sender and/or notificati...@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>



--
Regards,
Premal Shah.

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: HBase : get(...) vs scan and in-memory table

[Vladimir Rodionov]

There is no guarantee that your tables are in memory and you can not verify 
this directly. HBase will do its best to keep them in memory but its not 100%.
Cache is divided in 3 zones (default cache) and for IN_MEMORY tables HBase 
allocates 25% of a cache. If your data does not fit into this 25%
- try increasing block cache size.

>>Is the from-memory or from-disk read transparent to the client?

Yes, absolutely transparent.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Omkar Joshi [omkar.jo...@lntinfotech.com]
Sent: Wednesday, September 11, 2013 4:41 AM
To: user@hbase.apache.org
Subject: RE: HBase : get(...) vs scan and in-memory table

Hi JM,

Yes, I have DistributedCache on my mind too but not sure if those tables will 
be read-only in future. Besides, I want to check whether with their current 
size, those can be kept in-memory in HBase.

Regards,
Omkar Joshi



-Original Message-
From: Jean-Marc Spaggiari [mailto:jean-m...@spaggiari.org]
Sent: Wednesday, September 11, 2013 5:06 PM
To: user
Subject: Re: HBase : get(...) vs scan and in-memory table

Hi Omkar,

Your tables T1 and T2 are not so big. are your 100% they can fit in memory?
If yes, then why did you not distribute them to all the nodes in your MR
setup, like on a map format, using distributed cache? Then on your map
code, you will be 100% sure that both tables are local and in memory...

JM


2013/9/11 Omkar Joshi 

> I'm executing MR over HBase.
> The business logic in the reducer heavily accesses two tables, say T1(40k
> rows) and T2(90k rows). Currently, I'm executing the following steps :
> 1.In the constructor of the reducer class, doing something like this :
> HBaseCRUD hbaseCRUD = new HBaseCRUD();
>
> HTableInterface t1= hbaseCRUD.getTable("T1",
> "CF1", null, "C1", "C2");
> HTableInterface t2= hbaseCRUD.getTable("T2",
> "CF1", null, "C1", "C2");
> In the reduce(...)
>  String lowercase = ;
>
> /* Start : HBase code */
> /*
> * TRY using get(...) on the table rather than a
> * Scan!
> */
> Scan scan = new Scan();
> scan.setStartRow(lowercase.getBytes());
> scan.setStopRow(lowercase.getBytes());
>
> /*scan will return a single row*/
> ResultScanner resultScanner = t1.getScanner(scan);
>
> for (Result result : resultScanner) {
> /*business logic*/
> }
> Though not sure if the above code is sensible in first place, I have a
> question - would a get(...) provide any performance benefit over the scan?
> Get get = new Get(lowercase.getBytes());
> Result getResult = t1.get(get);
> Since T1 and T2 will be read-only(mostly), I think if kept in-memory, the
> performance will improve. As per HBase doc., I will have to re-create the
> tables T1 and T2. Please verify the correctness of my understanding :
> public void createTables(String tableName, boolean readOnly,
> boolean blockCacheEnabled, boolean inMemory,
> String... columnFamilyNames) throws IOException {
> // TODO Auto-generated method stub
>
> HTableDescriptor tableDesc = new HTableDescriptor(tableName);
> /* not sure !!! */
> tableDesc.setReadOnly(readOnly);
>
> HColumnDescriptor columnFamily = null;
>
> if (!(columnFamilyNames == null || columnFamilyNames.length == 0))
> {
>
> for (String columnFamilyName : columnFamilyNames) {
>
> columnFamily = new HColumnDescriptor(columnFamilyName);
> /*
>  * Start : Do these steps ensure that the column
>  * family(actually, the column data) is in-memory???
>  */
> columnFamily.setBlockCacheEnabled(blockCacheEnabled);
> columnFamily.setInMemory(inMemory);
> /*
>  * End : Do these steps ensure that the column
> family(actually,
>  * the column data) is in-memory???
>  */
>
> tableDesc.addFamily(columnFamily);
> }
> }
>
> hbaseAdmin.createTable(tableDesc);
> hbaseAdmin.close();
> }
> Once done :
>
>  1.  How to verify that the columns are in-memory and accessed from there
> and not the disk?
>  2.  Is the from-memory or from-disk read transparent to the client? In
> simple words, do I need to change the HTable access code in my reducer
> class? If yes, what are the changes?
>
>
> Regards,
> Omkar Joshi
>
>
>
> 
> The contents of this e-mail and any attachment(s) ma

RE: HBase Negation or NOT operator

[Vladimir Rodionov]

https://github.com/forcedotcom/phoenix to your help. It is worth taking a look 
at least.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Ashwin Jain [ashvyn.j...@gmail.com]
Sent: Tuesday, September 17, 2013 1:34 AM
To: user@hbase.apache.org
Subject: HBase Negation or NOT operator

Hello All,

Does HBase not support an SQL NOT operator on complex filters? I would like
to filter out whatever matches a complex nested filter.

my use case is to parse a query like this(below) and build a HBase filter
from it.
(field1=value1 AND NOT ((field2=value2 OR field3=value3) AND field4=value4))

How to go about this , any ideas?  What will be a better approach -
implement a custom filter that excludes a row qualified by another filter
or to convert input query into an opposite query.

Thanks,
Ashwin

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: hbase schema design

[Vladimir Rodionov]

> Is there a built-in functionality to generate (integer) surrogate values in
> hbase that can be used on the rowkey or does it need to be hand code it
> from scratch?

There is no such functionality in HBase. What are asking for is known as a 
dictionary compression :
unique 1-1 association between arbitrary strings and numeric values.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Ted Yu [yuzhih...@gmail.com]
Sent: Tuesday, September 17, 2013 9:53 AM
To: user@hbase.apache.org
Subject: Re: hbase schema design

I guess you were referring to section 6.3.2

bq. rowkey is stored and/ or read for every cell value

The above is true.

bq. the event description is a string of 0.1 to 2Kb

You can enable Data Block encoding to reduce storage.

Cheers



On Tue, Sep 17, 2013 at 9:44 AM, Adrian CAPDEFIER wrote:

> Howdy all,
>
> I'm trying to use hbase for the first time (plenty of other experience with
> RDBMS database though), and I have a couple of questions after reading The
> Book.
>
> I am a bit confused by the advice to reduce "the row size" in the hbase
> book. It states that every cell value is accomplished by the coordinates
> (row, column and timestamp). I'm just trying to be thorough, so am I to
> understand that the rowkey is stored and/ or read for every cell value in a
> record or just once per column family in a record?
>
> I am intrigued by the rows as columns design as described in the book at
> http://hbase.apache.org/book.html#rowkey.design. To make a long story
> short, I will end up with a table to store event types and number of
> occurrences in each day. I would prefer to have the event description as
> the row key and the dates when it happened as columns - up to 7300 for
> roughly 20 years.
> However, the event description is a string of 0.1 to 2Kb and if it is
> stored for each cell value, I will need to use a surrogate (shorter) value.
>
> Is there a built-in functionality to generate (integer) surrogate values in
> hbase that can be used on the rowkey or does it need to be hand code it
> from scratch?
>

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: HBase Table Row Count Optimization - A Solicitation For Help

[Vladimir Rodionov]

How long does it take for RowCounter Job for largest table to finish on your 
cluster?

Just curious.

On your options:

1. Not worth it probably - you may overload your cluster
2. Not sure this one differs from 1. Looks the same to me but more complex.
3. The same as 1 and 2

Counting rows in efficient way can be done if you sacrifice some accuracy :

http://highscalability.com/blog/2012/4/5/big-data-counting-how-to-count-a-billion-distinct-objects-us.html

Yeah, you will need coprocessors for that.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: James Birchfield [jbirchfi...@stumbleupon.com]
Sent: Friday, September 20, 2013 3:50 PM
To: user@hbase.apache.org
Subject: Re: HBase Table Row Count Optimization - A Solicitation For Help

Hadoop 2.0.0-cdh4.3.1

HBase 0.94.6-cdh4.3.1

110 servers, 0 dead, 238.2364 average load

Some other info, not sure if it helps or not.

Configured Capacity: 1295277834158080 (1.15 PB)
Present Capacity: 1224692609430678 (1.09 PB)
DFS Remaining: 624376503857152 (567.87 TB)
DFS Used: 600316105573526 (545.98 TB)
DFS Used%: 49.02%
Under replicated blocks: 0
Blocks with corrupt replicas: 1
Missing blocks: 0

It is hitting a production cluster, but I am not really sure how to calculate 
the load placed on the cluster.
On Sep 20, 2013, at 3:19 PM, Ted Yu  wrote:

> How many nodes do you have in your cluster ?
>
> When counting rows, what other load would be placed on the cluster ?
>
> What is the HBase version you're currently using / planning to use ?
>
> Thanks
>
>
> On Fri, Sep 20, 2013 at 2:47 PM, James Birchfield <
> jbirchfi...@stumbleupon.com> wrote:
>
>>After reading the documentation and scouring the mailing list
>> archives, I understand there is no real support for fast row counting in
>> HBase unless you build some sort of tracking logic into your code.  In our
>> case, we do not have such logic, and have massive amounts of data already
>> persisted.  I am running into the issue of very long execution of the
>> RowCounter MapReduce job against very large tables (multi-billion for many
>> is our estimate).  I understand why this issue exists and am slowly
>> accepting it, but I am hoping I can solicit some possible ideas to help
>> speed things up a little.
>>
>>My current task is to provide total row counts on about 600
>> tables, some extremely large, some not so much.  Currently, I have a
>> process that executes the MapRduce job in process like so:
>>
>>Job job = RowCounter.createSubmittableJob(
>>ConfigManager.getConfiguration(),
>> new String[]{tableName});
>>boolean waitForCompletion =
>> job.waitForCompletion(true);
>>Counters counters = job.getCounters();
>>Counter rowCounter =
>> counters.findCounter(hbaseadminconnection.Counters.ROWS);
>>return rowCounter.getValue();
>>
>>At the moment, each MapReduce job is executed in serial order, so
>> counting one table at a time.  For the current implementation of this whole
>> process, as it stands right now, my rough timing calculations indicate that
>> fully counting all the rows of these 600 tables will take anywhere between
>> 11 to 22 days.  This is not what I consider a desirable timeframe.
>>
>>I have considered three alternative approaches to speed things up.
>>
>>First, since the application is not heavily CPU bound, I could use
>> a ThreadPool and execute multiple MapReduce jobs at the same time looking
>> at different tables.  I have never done this, so I am unsure if this would
>> cause any unanticipated side effects.
>>
>>Second, I could distribute the processes.  I could find as many
>> machines that can successfully talk to the desired cluster properly, give
>> them a subset of tables to work on, and then combine the results post
>> process.
>>
>>Third, I could combine both the above approaches and run a
>> distributed set of multithreaded process to execute the MapReduce jobs in
>> parallel.
>>
>>Although it seems to have been asked and answered many times, I
>> will ask once again.  Without the need to change our current configurations
>> or restart the clusters, is there a faster approach to obtain row counts?
>> FYI, my cache size for the Scan is set to 1000.  I have experimented with
>> different numbers, but nothing made a noticeable difference.  Any advice or
>> feedback would be greatly appreciated!
>

RE: Large Files in Column Qualifier

[Vladimir Rodionov]

HBase is not  a file storage. It was not designed to be a file storage. 
Depending on your usage pattern I would suggest you another approach:

Store your files in a large "upload bundles"  on HDFS. You will need a 
collector(s) process for that.  Store references (Upload file name, offset and 
size)
in HBase.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Geovanie Marquez [geovanie.marq...@gmail.com]
Sent: Saturday, September 21, 2013 6:05 AM
To: user@hbase.apache.org
Subject: Large Files in Column Qualifier

I am evaluating an HBase design that would require that it rarely house a
1GB file in the column qualifier. Files range from 1GB - 1KB. These files
are raw files being ingested from clients and to be kept for some period of
time (several years) for quality control purposes. The application does not
depend on these files being in HBase, the files would be used by QA
personnel for data forensics to find out why data behaved unexpectedly in
the app or in our QC processes. That being said a lot of the reasons I've
read for not maintaining the data in HBase doesn't apply: compaction
storms, or performance degradation, since we can throttle how we place the
data in here.

I'd like to use HBase because it offers potential for indexing the data
later and potential for total data population analysis over solutions
involving HDFS as well as the use case where we receive tiny KB files more
often than not which would contribute to the Namenodes memory restrictions.
I could HAR these in HDFS but then indexing and more flexible options for
data analysis go out the window.

Does anyone see some glaring oversight I may be making in this design
consideration?

Thanks for your time.

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: Still not sure who to blame: Hbase, zookeeper, security?

[Vladimir Rodionov]

Version?

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Jay Vyas [jayunit...@gmail.com]
Sent: Friday, September 27, 2013 11:14 AM
To: user@hbase.apache.org
Subject: Still not sure who to blame: Hbase, zookeeper, security?

A few days ago I pasted logs from my cluster which wont let me create a
table:

http://stackoverflow.com/questions/18993200/hbase-cant-create-table-who-to-blame-hmaster-or-zookeeper

Even though hmaster is running according to JPS, I get a  please hold
exception.

Hate to bump, but I think it would be good to have an answer on
stackoverflow, or at least some comments.  Im sure others have or will run
into this error someday.

In any case, once its working, I'll paste an answer.

At the moment Im still wondering, how do we debug the master is
initializing, PleaseWaitException?
--
Jay Vyas
http://jayunit100.blogspot.com

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: Coprocessor Errors

[Vladimir Rodionov]

Guava lib issue (more than one in a CLASSPATH)?

http://code.google.com/p/guava-libraries/issues/detail?id=1318

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Ted Yu [yuzhih...@gmail.com]
Sent: Monday, September 30, 2013 3:10 PM
To: user@hbase.apache.org
Subject: Re: Coprocessor Errors

bq. unable to read call parameters: class
com.google.common.collect.AbstractMapBasedMultimap
overrides final method setMap.(Ljava/util/Map;)V

Can you show us the classpath ?

Thanks


On Mon, Sep 30, 2013 at 3:01 PM, Young Kim  wrote:

> Hello,
>
> I've been running into this odd error when running a coprocessor on
> hbase-0.94.6:
>
> 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
> 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client environment:host.name
> =templar1
> 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> environment:java.version=1.7.0_21
> 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> environment:java.vendor=Oracle Corporation
> 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> environment:java.home=/usr/java/jdk1.7.0_21/jre
> 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> environment:java.class.path=/opt/content/hbase-coproc-cr-current.jar
> 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> environment:java.io.tmpdir=/tmp
> 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> environment:java.compiler=
> 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client environment:os.name
> =Linux
> 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> environment:os.arch=amd64
> 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> environment:os.version=2.6.32-279.19.1.el6.x86_64
> 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client environment:user.name
> =croll
> 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> environment:user.home=/home/croll
> 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> environment:user.dir=/home/ykim
> 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Initiating client connection,
> connectString=archon2:2181,archon1:2181,templar2:2181,templar1:2181,archon3:2181
> sessionTimeout=18 watcher=hconnection
> 13/09/30 14:53:35 INFO zookeeper.RecoverableZooKeeper: The identifier of
> this process is 10...@templar1.int.crunchyroll.com
> 13/09/30 14:53:35 INFO zookeeper.ClientCnxn: Opening socket connection to
> server templar1/10.0.9.1:2181. Will not attempt to authenticate using
> SASL (unknown error)
> 13/09/30 14:53:35 INFO zookeeper.ClientCnxn: Socket connection established
> to templar1/10.0.9.1:2181, initiating session
> 13/09/30 14:53:35 INFO zookeeper.ClientCnxn: Session establishment
> complete on server templar1/10.0.9.1:2181, sessionid = 0x140c1b8d94c0c95,
> negotiated timeout = 4
> 13/09/30 14:54:52 WARN
> client.HConnectionManager$HConnectionImplementation: Error executing for row
> java.util.concurrent.ExecutionException:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=10, exceptions:
> Mon Sep 30 14:54:10 PDT 2013,
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@390e9a7d,
> org.apache.hadoop.ipc.RemoteException: IPC server unable to read call
> parameters: class com.google.common.collect.AbstractMapBasedMultimap
> overrides final method setMap.(Ljava/util/Map;)V
> Mon Sep 30 14:54:12 PDT 2013,
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@390e9a7d,
> org.apache.hadoop.ipc.RemoteException: IPC server unable to read call
> parameters: class com.google.common.collect.AbstractMapBasedMultimap
> overrides final method setMap.(Ljava/util/Map;)V
> Mon Sep 30 14:54:14 PDT 2013,
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@390e9a7d,
> org.apache.hadoop.ipc.RemoteException: IPC server unable to read call
> parameters: class com.google.common.collect.AbstractMapBasedMultimap
> overrides final method setMap.(Ljava/util/Map;)V
> Mon Sep 30 14:54:15 PDT 2013,
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@390e9a7d,
> org.apache.hadoop.ipc.RemoteException: IPC server unable to read call
> parameters: class com.google.common.collect.AbstractMapBasedMultimap
> overrides final method setMap.(Ljava/util/Map;)V
> Mon Sep 30 14:54:17 PDT 2013,
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@390e9a7d,
> org.apache.hadoop.ipc.RemoteException: IPC server unable to read call
> parameters: class com.google.common.collect.AbstractMapBasedMultimap
> overrides final method setMap.(Ljava/util/Map;)V
> Mon Sep 30 14:54:20 PDT 2013,
> org.apache.hadoop.hbase.ipc.ExecRP

RE: Coprocessor Errors

[Vladimir Rodionov]

Yes, you have to restart servers.
FYI, the right version of guava - guava-11.0.2.jar

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Young Kim [y...@crunchyroll.com]
Sent: Monday, September 30, 2013 4:57 PM
To: user@hbase.apache.org
Subject: Re: Coprocessor Errors

Thanks for the feedback. That is indeed what it looks like. I just looked under 
/usr/lib/hbase/lib, and there were two instances of guava jars in our region 
and master servers. After removing the deprecated one, it seems like this error 
is still popping up. Does anyone know if a restart is required for the changes 
to take affect?


On Monday, September 30, 2013 at 4:28 PM, Vladimir Rodionov wrote:

> Guava lib issue (more than one in a CLASSPATH)?
>
> http://code.google.com/p/guava-libraries/issues/detail?id=1318
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com (http://www.carrieriq.com)
> e-mail: vrodio...@carrieriq.com (mailto:vrodio...@carrieriq.com)
>
> 
> From: Ted Yu [yuzhih...@gmail.com (mailto:yuzhih...@gmail.com)]
> Sent: Monday, September 30, 2013 3:10 PM
> To: user@hbase.apache.org (mailto:user@hbase.apache.org)
> Subject: Re: Coprocessor Errors
>
> bq. unable to read call parameters: class
> com.google.common.collect.AbstractMapBasedMultimap
> overrides final method setMap.(Ljava/util/Map;)V
>
> Can you show us the classpath ?
>
> Thanks
>
>
> On Mon, Sep 30, 2013 at 3:01 PM, Young Kim  (mailto:y...@crunchyroll.com)> wrote:
>
> > Hello,
> >
> > I've been running into this odd error when running a coprocessor on
> > hbase-0.94.6:
> >
> > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
> > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client environment:host.name 
> > (http://host.name)
> > =templar1
> > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > environment:java.version=1.7.0_21
> > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > environment:java.vendor=Oracle Corporation
> > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > environment:java.home=/usr/java/jdk1.7.0_21/jre
> > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > environment:java.class.path=/opt/content/hbase-coproc-cr-current.jar
> > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > environment:java.io.tmpdir=/tmp
> > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > environment:java.compiler=
> > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client environment:os.name 
> > (http://os.name)
> > =Linux
> > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > environment:os.arch=amd64
> > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > environment:os.version=2.6.32-279.19.1.el6.x86_64
> > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client environment:user.name 
> > (http://user.name)
> > =croll
> > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > environment:user.home=/home/croll
> > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > environment:user.dir=/home/ykim
> > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Initiating client connection,
> > connectString=archon2:2181,archon1:2181,templar2:2181,templar1:2181,archon3:2181
> > sessionTimeout=18 watcher=hconnection
> > 13/09/30 14:53:35 INFO zookeeper.RecoverableZooKeeper: The identifier of
> > this process is 10...@templar1.int.crunchyroll.com 
> > (mailto:10...@templar1.int.crunchyroll.com)
> > 13/09/30 14:53:35 INFO zookeeper.ClientCnxn: Opening socket connection to
> > server templar1/10.0.9.1:2181. Will not attempt to authenticate using
> > SASL (unknown error)
> > 13/09/30 14:53:35 INFO zookeeper.ClientCnxn: Socket connection established
> > to templar1/10.0.9.1:2181, initiating session
> > 13/09/30 14:53:35 INFO zookeeper.ClientCnxn: Session establishment
> > complete on server templar1/10.0.9.1:2181, sessionid = 0x140c1b8d94c0c95,
> > negotiated timeout = 4
> > 13/09/30 14:54:52 WARN
> > client.HConnectionManager$HConnectionImplementation: Error executing for row
> > java.util.concurrent.ExecutionException:
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> > attempts=10, exceptions:
> > Mon Sep 30 14:54:10 PDT 2013,
> > 

RE: Coprocessor Errors

[Vladimir Rodionov]

No backward API compatibility, for example. 

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Young Kim [y...@crunchyroll.com]
Sent: Monday, September 30, 2013 6:04 PM
To: user@hbase.apache.org
Subject: Re: Coprocessor Errors

Is there a reason why it has to be guava-11.0.2? We're trying to use 
guava-14.0.1.

On Monday, September 30, 2013 at 5:59 PM, Vladimir Rodionov wrote:

> Yes, you have to restart servers.
> FYI, the right version of guava - guava-11.0.2.jar
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com (http://www.carrieriq.com)
> e-mail: vrodio...@carrieriq.com (mailto:vrodio...@carrieriq.com)
>
> 
> From: Young Kim [y...@crunchyroll.com (mailto:y...@crunchyroll.com)]
> Sent: Monday, September 30, 2013 4:57 PM
> To: user@hbase.apache.org (mailto:user@hbase.apache.org)
> Subject: Re: Coprocessor Errors
>
> Thanks for the feedback. That is indeed what it looks like. I just looked 
> under /usr/lib/hbase/lib, and there were two instances of guava jars in our 
> region and master servers. After removing the deprecated one, it seems like 
> this error is still popping up. Does anyone know if a restart is required for 
> the changes to take affect?
>
>
> On Monday, September 30, 2013 at 4:28 PM, Vladimir Rodionov wrote:
>
> > Guava lib issue (more than one in a CLASSPATH)?
> >
> > http://code.google.com/p/guava-libraries/issues/detail?id=1318
> >
> > Best regards,
> > Vladimir Rodionov
> > Principal Platform Engineer
> > Carrier IQ, www.carrieriq.com (http://www.carrieriq.com)
> > e-mail: vrodio...@carrieriq.com (mailto:vrodio...@carrieriq.com)
> >
> > 
> > From: Ted Yu [yuzhih...@gmail.com (mailto:yuzhih...@gmail.com)]
> > Sent: Monday, September 30, 2013 3:10 PM
> > To: user@hbase.apache.org (mailto:user@hbase.apache.org)
> > Subject: Re: Coprocessor Errors
> >
> > bq. unable to read call parameters: class
> > com.google.common.collect.AbstractMapBasedMultimap
> > overrides final method setMap.(Ljava/util/Map;)V
> >
> > Can you show us the classpath ?
> >
> > Thanks
> >
> >
> > On Mon, Sep 30, 2013 at 3:01 PM, Young Kim  > (mailto:y...@crunchyroll.com)> wrote:
> >
> > > Hello,
> > >
> > > I've been running into this odd error when running a coprocessor on
> > > hbase-0.94.6:
> > >
> > > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > > environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
> > > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client environment:host.name 
> > > (http://host.name)
> > > =templar1
> > > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > > environment:java.version=1.7.0_21
> > > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > > environment:java.vendor=Oracle Corporation
> > > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > > environment:java.home=/usr/java/jdk1.7.0_21/jre
> > > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > > environment:java.class.path=/opt/content/hbase-coproc-cr-current.jar
> > > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> > > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > > environment:java.io.tmpdir=/tmp
> > > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > > environment:java.compiler=
> > > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client environment:os.name 
> > > (http://os.name)
> > > =Linux
> > > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > > environment:os.arch=amd64
> > > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > > environment:os.version=2.6.32-279.19.1.el6.x86_64
> > > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client environment:user.name 
> > > (http://user.name)
> > > =croll
> > > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > > environment:user.home=/home/croll
> > > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > > environment:user.dir=/home/ykim
> > > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Initiating client connection,
> > > connectString=archon2:2181,archon1:2181,templar2:2181,templar1:2181,archon3:2181
> > > sessionTimeout=18 watcher=hconnection
>

RE: Coprocessor Errors

[Vladimir Rodionov]

HBase 0.94.x is not compatible with the latest guava: 

class com.google.common.collect.AbstractMapBasedMultimap
> > > > overrides final method setMap.(Ljava/util/Map;)V

AbstractMapBasedMultimap class hierarchy has been changed in 14 version, that 
is why you've got these errors.
You have two choices :

1. keep 14.0.1

Pro: the latest Guava API.
Contra: HBase does not work

2. keep 11.0.2

Pro: HBase works
Contra: not the latest Guava API 


Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Young Kim [y...@crunchyroll.com]
Sent: Monday, September 30, 2013 6:30 PM
To: user@hbase.apache.org
Subject: Re: Coprocessor Errors

Is there something specific that uses guava-11.0.2? We're reliant on 14.0.1 due 
to some new data structures presented in guava.


On Monday, September 30, 2013 at 6:22 PM, Vladimir Rodionov wrote:

> No backward API compatibility, for example.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com (http://www.carrieriq.com)
> e-mail: vrodio...@carrieriq.com (mailto:vrodio...@carrieriq.com)
>
> 
> From: Young Kim [y...@crunchyroll.com (mailto:y...@crunchyroll.com)]
> Sent: Monday, September 30, 2013 6:04 PM
> To: user@hbase.apache.org (mailto:user@hbase.apache.org)
> Subject: Re: Coprocessor Errors
>
> Is there a reason why it has to be guava-11.0.2? We're trying to use 
> guava-14.0.1.
>
> On Monday, September 30, 2013 at 5:59 PM, Vladimir Rodionov wrote:
>
> > Yes, you have to restart servers.
> > FYI, the right version of guava - guava-11.0.2.jar
> >
> > Best regards,
> > Vladimir Rodionov
> > Principal Platform Engineer
> > Carrier IQ, www.carrieriq.com (http://www.carrieriq.com)
> > e-mail: vrodio...@carrieriq.com (mailto:vrodio...@carrieriq.com)
> >
> > 
> > From: Young Kim [y...@crunchyroll.com (mailto:y...@crunchyroll.com)]
> > Sent: Monday, September 30, 2013 4:57 PM
> > To: user@hbase.apache.org (mailto:user@hbase.apache.org)
> > Subject: Re: Coprocessor Errors
> >
> > Thanks for the feedback. That is indeed what it looks like. I just looked 
> > under /usr/lib/hbase/lib, and there were two instances of guava jars in our 
> > region and master servers. After removing the deprecated one, it seems like 
> > this error is still popping up. Does anyone know if a restart is required 
> > for the changes to take affect?
> >
> >
> > On Monday, September 30, 2013 at 4:28 PM, Vladimir Rodionov wrote:
> >
> > > Guava lib issue (more than one in a CLASSPATH)?
> > >
> > > http://code.google.com/p/guava-libraries/issues/detail?id=1318
> > >
> > > Best regards,
> > > Vladimir Rodionov
> > > Principal Platform Engineer
> > > Carrier IQ, www.carrieriq.com (http://www.carrieriq.com)
> > > e-mail: vrodio...@carrieriq.com (mailto:vrodio...@carrieriq.com)
> > >
> > > 
> > > From: Ted Yu [yuzhih...@gmail.com (mailto:yuzhih...@gmail.com)]
> > > Sent: Monday, September 30, 2013 3:10 PM
> > > To: user@hbase.apache.org (mailto:user@hbase.apache.org)
> > > Subject: Re: Coprocessor Errors
> > >
> > > bq. unable to read call parameters: class
> > > com.google.common.collect.AbstractMapBasedMultimap
> > > overrides final method setMap.(Ljava/util/Map;)V
> > >
> > > Can you show us the classpath ?
> > >
> > > Thanks
> > >
> > >
> > > On Mon, Sep 30, 2013 at 3:01 PM, Young Kim  > > (mailto:y...@crunchyroll.com)> wrote:
> > >
> > > > Hello,
> > > >
> > > > I've been running into this odd error when running a coprocessor on
> > > > hbase-0.94.6:
> > > >
> > > > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > > > environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 
> > > > GMT
> > > > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client 
> > > > environment:host.name (http://host.name)
> > > > =templar1
> > > > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > > > environment:java.version=1.7.0_21
> > > > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > > > environment:java.vendor=Oracle Corporation
> > > > 13/09/30 14:53:35 INFO zookeeper.ZooKeeper: Client
> > > > environment:java.home=/usr/java/jdk1.7.0_21/jre
>

RE: HBase Random Read latency > 100ms

[Vladimir Rodionov]

Ramu, your HBase configuration (128GB of heap) is far from optimal.
Nobody runs HBase with that amount of heap to my best knowledge.
32GB of RAM is the usual upper limit. We run 8-12GB in production.

What else, your IO capacity is VERY low. 2 SATA drives in RAID 1 for mostly 
random reads load?
You should have 8, better 12-16 drives per server. Forget about RAID. You have 
HDFS.

Block cache in your case does not help much , as since your read amplification 
is at least x20 (16KB block and 724 B read) - its just waste
RAM (heap). In your case you do not need LARGE heap and LARGE block cache.

I advise you reconsidering your hardware spec, applying all optimizations 
mentioned already in this thread and lowering your expectations.

With a right hardware you will be able to get 500-1000 truly random reads per 
server.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Ramu M S [ramu.ma...@gmail.com]
Sent: Monday, October 07, 2013 5:23 AM
To: user@hbase.apache.org
Subject: Re: HBase Random Read latency > 100ms

Hi Bharath,

I am little confused about the metrics displayed by Cloudera. Even when
there are no oeprations, the gc_time metric is showing 2s constant in the
graph. Is this the CMS gc_time (in that case no JVm pause) or the GC pause.

GC timings reported earlier is the average taken for gc_time metric for all
region servers.

Regards,
Ramu


On Mon, Oct 7, 2013 at 9:10 PM, Ramu M S  wrote:

> Jean,
>
> Yes. It is 2 drives.
>
> - Ramu
>
>
> On Mon, Oct 7, 2013 at 8:45 PM, Jean-Marc Spaggiari <
> jean-m...@spaggiari.org> wrote:
>
>> Quick questionon the disk side.
>>
>> When you say:
>> 800 GB SATA (7200 RPM) Disk
>> Is it 1x800GB? It's raid 1, so might be 2 drives? What's the
>> configuration?
>>
>> JM
>>
>>
>> 2013/10/7 Ramu M S 
>>
>> > Lars, Bharath,
>> >
>> > Compression is disabled for the table. This was not intended from the
>> > evaluation.
>> > I forgot to mention that during table creation. I will enable snappy
>> and do
>> > major compaction again.
>> >
>> > Please suggest other options to try out and also suggestions for the
>> > previous questions.
>> >
>> > Thanks,
>> > Ramu
>> >
>> >
>> > On Mon, Oct 7, 2013 at 6:35 PM, Ramu M S  wrote:
>> >
>> > > Bharath,
>> > >
>> > > I was about to report this. Yes indeed there is too much of GC time.
>> > > Just verified the GC time using Cloudera Manager statistics(Every
>> minute
>> > > update).
>> > >
>> > > For each Region Server,
>> > >  - During Read: Graph shows 2s constant.
>> > >  - During Compaction: Graph starts with 7s and goes as high as 20s
>> during
>> > > end.
>> > >
>> > > Few more questions,
>> > > 1. For the current evaluation, since the reads are completely random
>> and
>> > I
>> > > don't expect to read same data again can I set the Heap to the
>> default 1
>> > GB
>> > > ?
>> > >
>> > > 2. Can I completely turn off BLOCK CACHE for this table?
>> > > http://hbase.apache.org/book/regionserver.arch.html recommends
>> that
>> > > for Randm reads.
>> > >
>> > > 3. But in the next phase of evaluation, We are interested to use
>> HBase as
>> > > In-memory KV DB by having the latest data in RAM (To the tune of
>> around
>> > 128
>> > > GB in each RS, we are setting up 50-100 Node Cluster). I am very
>> curious
>> > to
>> > > hear any suggestions in this regard.
>> > >
>> > > Regards,
>> > > Ramu
>> > >
>> > >
>> > > On Mon, Oct 7, 2013 at 5:50 PM, Bharath Vissapragada <
>> > > bhara...@cloudera.com> wrote:
>> > >
>> > >> Hi Ramu,
>> > >>
>> > >> Thanks for reporting the results back. Just curious if you are
>> hitting
>> > any
>> > >> big GC pauses due to block cache churn on such large heap. Do you see
>> > it ?
>> > >>
>> > >> - Bharath
>> > >>
>> > >>
>> > >> On Mon, Oct 7, 2013 at 1:42 PM, Ramu M S 
>> wrote:
>> > >>
>> > >> > Lars,
>> > >> >
>> > >> > After changing the BLOCKSIZE to 16KB, the latency has reduced a
>> > little.
>

RE: HBase Random Read latency > 100ms

[Vladimir Rodionov]

Ramu,

If your working set of data fits into 192GB you may get additional boost by 
utilizing OS page cache, or wait until
0.98 release which introduces new bucket cache implementation (port of Facebook 
L2 cache). You can try vanilla bucket cache in 0.96 (not released yet
but is due soon). Both caches stores data off-heap, but Facebook version can 
store encoded and compressed data and vanilla bucket cache does not.
There are some options how to utilize efficiently available RAM (at least in 
upcoming HBase releases)
. If your data set does not fit RAM then your only hope is your 24 SAS drives. 
Depending on your RAID settings, disk IO perf, HDFS configuration (I think the 
latest Hadoop is preferable here).

OS page cache is most vulnerable and volatile, it can not be controlled and can 
be easily polluted by either some other processes or by HBase itself (long 
scan).
With Block cache you have more control but the first truly usable *official* 
implementation is going to be a part of 0.98 release.

As far as I understand, your use case would definitely covered by something 
similar to BigTable ScanCache (RowCache) , but there is no such cache in HBase 
yet.
One major advantage of RowCache vs BlockCache (apart from being much more 
efficient in RAM usage) is resilience to Region compactions. Each minor Region 
compaction invalidates partially
Region's data in BlockCache and major compaction invalidates this Region's data 
completely. This is not the case with RowCache (would it be implemented).

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Ramu M S [ramu.ma...@gmail.com]
Sent: Monday, October 07, 2013 5:25 PM
To: user@hbase.apache.org
Subject: Re: HBase Random Read latency > 100ms

Vladimir,

Yes. I am fully aware of the HDD limitation and wrong configurations wrt
RAID.
Unfortunately, the hardware is leased from others for this work and I
wasn't consulted to decide the h/w specification for the tests that I am
doing now. Even the RAID cannot be turned off or set to RAID-0

Production system is according to the Hadoop needs (100 Nodes with 16 Core
CPU, 192 GB RAM, 24 X 600GB SAS Drives, RAID cannot be completely turned
off, so we are creating 1 Virtual Disk containing only 1 Physical Disk and
the VD RAID level set to* *RAID-0). These systems are still not available. If
you have any suggestion on the production setup, I will be glad to hear.

Also, as pointed out earlier, we are planning to use HBase also as an in
memory KV store to access the latest data.
That's why RAM was considered huge in this configuration. But looks like we
would run into more problems than any gains from this.

Keeping that aside, I was trying to get the maximum out of the current
cluster or as you said Is 500-1000 OPS the max I could get out of this
setup?

Regards,
Ramu



Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: HBase Random Read latency > 100ms

[Vladimir Rodionov]

What are your current heap and block cache sizes?

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Ramu M S [ramu.ma...@gmail.com]
Sent: Monday, October 07, 2013 10:55 PM
To: user@hbase.apache.org
Subject: Re: HBase Random Read latency > 100ms

Hi All,

Average Latency is still around 80ms.
I have done the following,

1. Enabled Snappy Compression
2. Reduce the HFile size to 8 GB

Should I attribute these results to bad Disk Configuration OR anything else
to investigate?

- Ramu


On Tue, Oct 8, 2013 at 10:56 AM, Ramu M S  wrote:

> Vladimir,
>
> Thanks for the Insights into Future Caching features. Looks very
> interesting.
>
> - Ramu
>
>
> On Tue, Oct 8, 2013 at 10:45 AM, Vladimir Rodionov <
> vrodio...@carrieriq.com> wrote:
>
>> Ramu,
>>
>> If your working set of data fits into 192GB you may get additional boost
>> by utilizing OS page cache, or wait until
>> 0.98 release which introduces new bucket cache implementation (port of
>> Facebook L2 cache). You can try vanilla bucket cache in 0.96 (not released
>> yet
>> but is due soon). Both caches stores data off-heap, but Facebook version
>> can store encoded and compressed data and vanilla bucket cache does not.
>> There are some options how to utilize efficiently available RAM (at least
>> in upcoming HBase releases)
>> . If your data set does not fit RAM then your only hope is your 24 SAS
>> drives. Depending on your RAID settings, disk IO perf, HDFS configuration
>> (I think the latest Hadoop is preferable here).
>>
>> OS page cache is most vulnerable and volatile, it can not be controlled
>> and can be easily polluted by either some other processes or by HBase
>> itself (long scan).
>> With Block cache you have more control but the first truly usable
>> *official* implementation is going to be a part of 0.98 release.
>>
>> As far as I understand, your use case would definitely covered by
>> something similar to BigTable ScanCache (RowCache) , but there is no such
>> cache in HBase yet.
>> One major advantage of RowCache vs BlockCache (apart from being much more
>> efficient in RAM usage) is resilience to Region compactions. Each minor
>> Region compaction invalidates partially
>> Region's data in BlockCache and major compaction invalidates this
>> Region's data completely. This is not the case with RowCache (would it be
>> implemented).
>>
>> Best regards,
>> Vladimir Rodionov
>> Principal Platform Engineer
>> Carrier IQ, www.carrieriq.com
>> e-mail: vrodio...@carrieriq.com
>>
>> 
>> From: Ramu M S [ramu.ma...@gmail.com]
>> Sent: Monday, October 07, 2013 5:25 PM
>> To: user@hbase.apache.org
>> Subject: Re: HBase Random Read latency > 100ms
>>
>> Vladimir,
>>
>> Yes. I am fully aware of the HDD limitation and wrong configurations wrt
>> RAID.
>> Unfortunately, the hardware is leased from others for this work and I
>> wasn't consulted to decide the h/w specification for the tests that I am
>> doing now. Even the RAID cannot be turned off or set to RAID-0
>>
>> Production system is according to the Hadoop needs (100 Nodes with 16 Core
>> CPU, 192 GB RAM, 24 X 600GB SAS Drives, RAID cannot be completely turned
>> off, so we are creating 1 Virtual Disk containing only 1 Physical Disk and
>> the VD RAID level set to* *RAID-0). These systems are still not
>> available. If
>> you have any suggestion on the production setup, I will be glad to hear.
>>
>> Also, as pointed out earlier, we are planning to use HBase also as an in
>> memory KV store to access the latest data.
>> That's why RAM was considered huge in this configuration. But looks like
>> we
>> would run into more problems than any gains from this.
>>
>> Keeping that aside, I was trying to get the maximum out of the current
>> cluster or as you said Is 500-1000 OPS the max I could get out of this
>> setup?
>>
>> Regards,
>> Ramu
>>
>>
>>
>> Confidentiality Notice:  The information contained in this message,
>> including any attachments hereto, may be confidential and is intended to be
>> read only by the individual or entity to whom this message is addressed. If
>> the reader of this message is not the intended recipient or an agent or
>> designee of the intended recipient, please note that any review, use,
>> disclosure or distribution of this message or its attachments, in any form,
>> is strictly proh

RE: HBase Random Read latency > 100ms

[Vladimir Rodionov]

This can be GC related.

128GB heap size,
51.2GB - BlockCache size (on heap)

Zipfian distribution of small objects (712B)

Results: extreme cache pollution and eviction rate. High eviction -> High GC
As far as I remember, LruBlockCache does not have real-time eviction and odes 
it in batches,
these batches will add another latency spikes.

Decrease heap, reduce block cache size (to minimum) and repeat test.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: lars hofhansl [la...@apache.org]
Sent: Monday, October 07, 2013 2:04 PM
To: user@hbase.apache.org
Subject: Re: HBase Random Read latency > 100ms

He still should not see 100ms latency. 20ms, sure. 100ms seems large; there are 
still 8 machines serving the requests.

I agree this spec is far from optimal, but there is still something odd here.


Ramu, this does not look like a GC issue. You'd see much larger (worst case) 
latencies if that were the case (dozens of seconds).
Are you using 40 client from 40 different machines? Or from 40 different 
processes on the same machine? Or 40 threads in the same process?

Thanks.

-- Lars



____
 From: Vladimir Rodionov 
To: "user@hbase.apache.org" 
Sent: Monday, October 7, 2013 11:02 AM
Subject: RE: HBase Random Read latency > 100ms


Ramu, your HBase configuration (128GB of heap) is far from optimal.
Nobody runs HBase with that amount of heap to my best knowledge.
32GB of RAM is the usual upper limit. We run 8-12GB in production.

What else, your IO capacity is VERY low. 2 SATA drives in RAID 1 for mostly 
random reads load?
You should have 8, better 12-16 drives per server. Forget about RAID. You have 
HDFS.

Block cache in your case does not help much , as since your read amplification 
is at least x20 (16KB block and 724 B read) - its just waste
RAM (heap). In your case you do not need LARGE heap and LARGE block cache.

I advise you reconsidering your hardware spec, applying all optimizations 
mentioned already in this thread and lowering your expectations.

With a right hardware you will be able to get 500-1000 truly random reads per 
server.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com



From: Ramu M S [ramu.ma...@gmail.com]
Sent: Monday, October 07, 2013 5:23 AM
To: user@hbase.apache.org
Subject: Re: HBase Random Read latency > 100ms

Hi Bharath,

I am little confused about the metrics displayed by Cloudera. Even when
there are no oeprations, the gc_time metric is showing 2s constant in the
graph. Is this the CMS gc_time (in that case no JVm pause) or the GC pause.

GC timings reported earlier is the average taken for gc_time metric for all
region servers.

Regards,
Ramu


On Mon, Oct 7, 2013 at 9:10 PM, Ramu M S  wrote:

> Jean,
>
> Yes. It is 2 drives.
>
> - Ramu
>
>
> On Mon, Oct 7, 2013 at 8:45 PM, Jean-Marc Spaggiari <
> jean-m...@spaggiari.org> wrote:
>
>> Quick questionon the disk side.
>>
>> When you say:
>> 800 GB SATA (7200 RPM) Disk
>> Is it 1x800GB? It's raid 1, so might be 2 drives? What's the
>> configuration?
>>
>> JM
>>
>>
>> 2013/10/7 Ramu M S 
>>
>> > Lars, Bharath,
>> >
>> > Compression is disabled for the table. This was not intended from the
>> > evaluation.
>> > I forgot to mention that during table creation. I will enable snappy
>> and do
>> > major compaction again.
>> >
>> > Please suggest other options to try out and also suggestions for the
>> > previous questions.
>> >
>> > Thanks,
>> > Ramu
>> >
>> >
>> > On Mon, Oct 7, 2013 at 6:35 PM, Ramu M S  wrote:
>> >
>> > > Bharath,
>> > >
>> > > I was about to report this. Yes indeed there is too much of GC time.
>> > > Just verified the GC time using Cloudera Manager statistics(Every
>> minute
>> > > update).
>> > >
>> > > For each Region Server,
>> > >  - During Read: Graph shows 2s constant.
>> > >  - During Compaction: Graph starts with 7s and goes as high as 20s
>> during
>> > > end.
>> > >
>> > > Few more questions,
>> > > 1. For the current evaluation, since the reads are completely random
>> and
>> > I
>> > > don't expect to read same data again can I set the Heap to the
>> default 1
>> > GB
>> > > ?
>> > >
>> > > 2. Can I completely turn off BLOCK CACHE for this table?
>> > >http://hbase.apache.org/

RE: HBase Random Read latency > 100ms

[Vladimir Rodionov]

Ramu,

You have 8 server boxes and 10 client. You have 40 requests in parallel - 5 per 
RS/DN?

You have 5 requests on random reads in a IO queue of your single RAID1. With 
avg read latency of 10 ms, 5 requests in queue will give us 30ms. Add some 
overhead
of HDFS + HBase and you will probably have your issue explained ?

Your bottleneck is your disk system, I think. When you serve most of requests 
from disks as in your large data set scenario, make sure you have adequate disk 
sub-system and
that it is configured properly. Block Cache and OS page can not help you in 
this case as working data set is larger than both caches.

Good performance numbers in small data set scenario are explained by the fact 
that data fits into OS page cache and Block Cache - you do not read data from 
disk even if
you disable block cache.


Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Ramu M S [ramu.ma...@gmail.com]
Sent: Tuesday, October 08, 2013 6:00 PM
To: user@hbase.apache.org
Subject: Re: HBase Random Read latency > 100ms

Hi All,

After few suggestions from the mails earlier I changed the following,

1. Heap Size to 16 GB
2. Block Size to 16KB
3. HFile size to 8 GB (Table now has 256 regions, 32 per server)
4. Data Locality Index is 100 in all RS

I have clients running in 10 machines, each with 4 threads. So total 40.
This is same in all tests.

Result:
   1. Average latency is still >100ms.
   2. Heap occupancy is around 2-2.5 GB in all RS

Few more tests carried out yesterday,

TEST 1: Small data set (100 Million records, each with 724 bytes).
===
Configurations:
1. Heap Size to 1 GB
2. Block Size to 16KB
3. HFile size to 1 GB (Table now has 128 regions, 16 per server)
4. Data Locality Index is 100 in all RS

I disabled Block Cache on the table, to make sure I read everything from
disk, most of the time.

Result:
   1. Average Latency is 8ms and throughput went up to 6K/Sec per RS.
   2. With Block Cache enabled again, I got average latency around 2ms
and throughput of 10K/Sec per RS.
   Heap occupancy around 650 MB
   3. Increased the Heap to 16GB, with Block Cache still enabled, I got
average latency around 1 ms and throughput 20K/Sec per RS
   Heap Occupancy around 2-2.5 GB in all RS

TEST 2: Large Data set (1.8 Billion records, each with 724 bytes)
==
Configurations:
1. Heap Size to 1 GB
2. Block Size to 16KB
3. HFile size to 1 GB (Table now has 2048 regions, 256 per server)
4. Data Locality Index is 100 in all RS

Result:
  1. Average Latency is > 500ms to start with and gradually decreases, but
even after around 100 Million reads it is still >100 ms
  2. Block Cache = TRUE/FALSE does not make any difference here. Even Heap
Size (1GB / 16GB) does not make any difference.
  3. Heap occupancy is around 2-2.5 GB under 16GB Heap and around 650 MB
under 1GB Heap.

GC Time in all of the scenarios is around 2ms/Second, as shown in the
Cloudera Manager.

Reading most of the items from Disk in less data scenario gives better
results and very low latencies.

Number of regions per RS and HFile size does make a huge difference in my
Cluster.
Keeping 100 Regions per RS as max(Most of the discussions suggest this),
should I restrict the HFile size to 1GB? and thus reducing the storage
capacity (From 700 GB to 100GB per RS)?

Please advice.

Thanks,
Ramu


On Wed, Oct 9, 2013 at 4:58 AM, Vladimir Rodionov
wrote:

> What are your current heap and block cache sizes?
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodio...@carrieriq.com
>
> 
> From: Ramu M S [ramu.ma...@gmail.com]
> Sent: Monday, October 07, 2013 10:55 PM
> To: user@hbase.apache.org
> Subject: Re: HBase Random Read latency > 100ms
>
> Hi All,
>
> Average Latency is still around 80ms.
> I have done the following,
>
> 1. Enabled Snappy Compression
> 2. Reduce the HFile size to 8 GB
>
> Should I attribute these results to bad Disk Configuration OR anything else
> to investigate?
>
> - Ramu
>
>
> On Tue, Oct 8, 2013 at 10:56 AM, Ramu M S  wrote:
>
> > Vladimir,
> >
> > Thanks for the Insights into Future Caching features. Looks very
> > interesting.
> >
> > - Ramu
> >
> >
> > On Tue, Oct 8, 2013 at 10:45 AM, Vladimir Rodionov <
> > vrodio...@carrieriq.com> wrote:
> >
> >> Ramu,
> >>
> >> If your working set of data fits into 192GB you may get additional boost
> >> by utilizing OS page cache, or wait until
> >> 0.98 release which introduces new bucket cache implementation (port of
> >> Facebook 

RE: HBase Random Read latency > 100ms

[Vladimir Rodionov]

I suggest you two additional tests on large dataset:

Run one client thread per server (8 max) with:
1. SCR enabled
2. SCR disabled


Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Vladimir Rodionov
Sent: Tuesday, October 08, 2013 7:24 PM
To: user@hbase.apache.org
Subject: RE: HBase Random Read latency > 100ms

Ramu,

You have 8 server boxes and 10 client. You have 40 requests in parallel - 5 per 
RS/DN?

You have 5 requests on random reads in a IO queue of your single RAID1. With 
avg read latency of 10 ms, 5 requests in queue will give us 30ms. Add some 
overhead
of HDFS + HBase and you will probably have your issue explained ?

Your bottleneck is your disk system, I think. When you serve most of requests 
from disks as in your large data set scenario, make sure you have adequate disk 
sub-system and
that it is configured properly. Block Cache and OS page can not help you in 
this case as working data set is larger than both caches.

Good performance numbers in small data set scenario are explained by the fact 
that data fits into OS page cache and Block Cache - you do not read data from 
disk even if
you disable block cache.


Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Ramu M S [ramu.ma...@gmail.com]
Sent: Tuesday, October 08, 2013 6:00 PM
To: user@hbase.apache.org
Subject: Re: HBase Random Read latency > 100ms

Hi All,

After few suggestions from the mails earlier I changed the following,

1. Heap Size to 16 GB
2. Block Size to 16KB
3. HFile size to 8 GB (Table now has 256 regions, 32 per server)
4. Data Locality Index is 100 in all RS

I have clients running in 10 machines, each with 4 threads. So total 40.
This is same in all tests.

Result:
   1. Average latency is still >100ms.
   2. Heap occupancy is around 2-2.5 GB in all RS

Few more tests carried out yesterday,

TEST 1: Small data set (100 Million records, each with 724 bytes).
===
Configurations:
1. Heap Size to 1 GB
2. Block Size to 16KB
3. HFile size to 1 GB (Table now has 128 regions, 16 per server)
4. Data Locality Index is 100 in all RS

I disabled Block Cache on the table, to make sure I read everything from
disk, most of the time.

Result:
   1. Average Latency is 8ms and throughput went up to 6K/Sec per RS.
   2. With Block Cache enabled again, I got average latency around 2ms
and throughput of 10K/Sec per RS.
   Heap occupancy around 650 MB
   3. Increased the Heap to 16GB, with Block Cache still enabled, I got
average latency around 1 ms and throughput 20K/Sec per RS
   Heap Occupancy around 2-2.5 GB in all RS

TEST 2: Large Data set (1.8 Billion records, each with 724 bytes)
==
Configurations:
1. Heap Size to 1 GB
2. Block Size to 16KB
3. HFile size to 1 GB (Table now has 2048 regions, 256 per server)
4. Data Locality Index is 100 in all RS

Result:
  1. Average Latency is > 500ms to start with and gradually decreases, but
even after around 100 Million reads it is still >100 ms
  2. Block Cache = TRUE/FALSE does not make any difference here. Even Heap
Size (1GB / 16GB) does not make any difference.
  3. Heap occupancy is around 2-2.5 GB under 16GB Heap and around 650 MB
under 1GB Heap.

GC Time in all of the scenarios is around 2ms/Second, as shown in the
Cloudera Manager.

Reading most of the items from Disk in less data scenario gives better
results and very low latencies.

Number of regions per RS and HFile size does make a huge difference in my
Cluster.
Keeping 100 Regions per RS as max(Most of the discussions suggest this),
should I restrict the HFile size to 1GB? and thus reducing the storage
capacity (From 700 GB to 100GB per RS)?

Please advice.

Thanks,
Ramu


On Wed, Oct 9, 2013 at 4:58 AM, Vladimir Rodionov
wrote:

> What are your current heap and block cache sizes?
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodio...@carrieriq.com
>
> 
> From: Ramu M S [ramu.ma...@gmail.com]
> Sent: Monday, October 07, 2013 10:55 PM
> To: user@hbase.apache.org
> Subject: Re: HBase Random Read latency > 100ms
>
> Hi All,
>
> Average Latency is still around 80ms.
> I have done the following,
>
> 1. Enabled Snappy Compression
> 2. Reduce the HFile size to 8 GB
>
> Should I attribute these results to bad Disk Configuration OR anything else
> to investigate?
>
> - Ramu
>
>
> On Tue, Oct 8, 2013 at 10:56 AM, Ramu M S  wrote:
>
> > Vladimir,
> >
> > Thanks for the Insights into Future Caching features. Looks very
> > interesting.
> >
> > - Ra

RE: HBase Random Read latency > 100ms

[Vladimir Rodionov]

Upd.

Each HBase Get = 2 HDFS read IO (index block + data block)= 4 File IO  (data + 
.crc) in a worst case. I think if Bloom Filter is enabled than
it is going to be 6 File IO in a worst case (large data set), therefore you 
will have not 5 IO requests in queue but up to 20-30 IO requests in a queue
This definitely explains > 100ms avg latency.



Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Vladimir Rodionov
Sent: Tuesday, October 08, 2013 7:24 PM
To: user@hbase.apache.org
Subject: RE: HBase Random Read latency > 100ms

Ramu,

You have 8 server boxes and 10 client. You have 40 requests in parallel - 5 per 
RS/DN?

You have 5 requests on random reads in a IO queue of your single RAID1. With 
avg read latency of 10 ms, 5 requests in queue will give us 30ms. Add some 
overhead
of HDFS + HBase and you will probably have your issue explained ?

Your bottleneck is your disk system, I think. When you serve most of requests 
from disks as in your large data set scenario, make sure you have adequate disk 
sub-system and
that it is configured properly. Block Cache and OS page can not help you in 
this case as working data set is larger than both caches.

Good performance numbers in small data set scenario are explained by the fact 
that data fits into OS page cache and Block Cache - you do not read data from 
disk even if
you disable block cache.


Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Ramu M S [ramu.ma...@gmail.com]
Sent: Tuesday, October 08, 2013 6:00 PM
To: user@hbase.apache.org
Subject: Re: HBase Random Read latency > 100ms

Hi All,

After few suggestions from the mails earlier I changed the following,

1. Heap Size to 16 GB
2. Block Size to 16KB
3. HFile size to 8 GB (Table now has 256 regions, 32 per server)
4. Data Locality Index is 100 in all RS

I have clients running in 10 machines, each with 4 threads. So total 40.
This is same in all tests.

Result:
   1. Average latency is still >100ms.
   2. Heap occupancy is around 2-2.5 GB in all RS

Few more tests carried out yesterday,

TEST 1: Small data set (100 Million records, each with 724 bytes).
===
Configurations:
1. Heap Size to 1 GB
2. Block Size to 16KB
3. HFile size to 1 GB (Table now has 128 regions, 16 per server)
4. Data Locality Index is 100 in all RS

I disabled Block Cache on the table, to make sure I read everything from
disk, most of the time.

Result:
   1. Average Latency is 8ms and throughput went up to 6K/Sec per RS.
   2. With Block Cache enabled again, I got average latency around 2ms
and throughput of 10K/Sec per RS.
   Heap occupancy around 650 MB
   3. Increased the Heap to 16GB, with Block Cache still enabled, I got
average latency around 1 ms and throughput 20K/Sec per RS
   Heap Occupancy around 2-2.5 GB in all RS

TEST 2: Large Data set (1.8 Billion records, each with 724 bytes)
==
Configurations:
1. Heap Size to 1 GB
2. Block Size to 16KB
3. HFile size to 1 GB (Table now has 2048 regions, 256 per server)
4. Data Locality Index is 100 in all RS

Result:
  1. Average Latency is > 500ms to start with and gradually decreases, but
even after around 100 Million reads it is still >100 ms
  2. Block Cache = TRUE/FALSE does not make any difference here. Even Heap
Size (1GB / 16GB) does not make any difference.
  3. Heap occupancy is around 2-2.5 GB under 16GB Heap and around 650 MB
under 1GB Heap.

GC Time in all of the scenarios is around 2ms/Second, as shown in the
Cloudera Manager.

Reading most of the items from Disk in less data scenario gives better
results and very low latencies.

Number of regions per RS and HFile size does make a huge difference in my
Cluster.
Keeping 100 Regions per RS as max(Most of the discussions suggest this),
should I restrict the HFile size to 1GB? and thus reducing the storage
capacity (From 700 GB to 100GB per RS)?

Please advice.

Thanks,
Ramu


On Wed, Oct 9, 2013 at 4:58 AM, Vladimir Rodionov
wrote:

> What are your current heap and block cache sizes?
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodio...@carrieriq.com
>
> 
> From: Ramu M S [ramu.ma...@gmail.com]
> Sent: Monday, October 07, 2013 10:55 PM
> To: user@hbase.apache.org
> Subject: Re: HBase Random Read latency > 100ms
>
> Hi All,
>
> Average Latency is still around 80ms.
> I have done the following,
>
> 1. Enabled Snappy Compression
> 2. Reduce the HFile size to 8 GB
>
> Should I attribute these results to bad Disk Configuration OR anything else
> to investigate?
>
> - 

RE: HBase Random Read latency > 100ms

[Vladimir Rodionov]

I can't say for SCR. There is a possibility that the feature is broken, of 
course.
But the fact that hbase.regionserver.checksum.verify does not affect 
performance means that OS caches
effectively HDFS checksum files.


Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Ramu M S [ramu.ma...@gmail.com]
Sent: Wednesday, October 09, 2013 12:11 AM
To: user@hbase.apache.org; lars hofhansl
Subject: Re: HBase Random Read latency > 100ms

Hi All,

Sorry. There was some mistake in the tests (Clients were not reduced,
forgot to change the parameter before running tests).

With 8 Clients and,

SCR Enabled : Average Latency is 25 ms, IO Wait % is around 8
SCR Disabled: Average Latency is 10 ms, IO Wait % is around 2

Still, SCR disabled gives better results, which confuse me. Can anyone
clarify?

Also, I tried setting the parameter (hbase.regionserver.checksum.verify as
true) Lars suggested with SCR disabled.
Average Latency is around 9.8 ms, a fraction lesser.

Thanks
Ramu


On Wed, Oct 9, 2013 at 3:32 PM, Ramu M S  wrote:

> Hi All,
>
> I just ran only 8 parallel clients,
>
> With SCR Enabled : Average Latency is 80 ms, IO Wait % is around 8
> With SCR Disabled: Average Latency is 40 ms, IO Wait % is around 2
>
> I always thought SCR enabled, allows a client co-located with the DataNode
> to read HDFS file blocks directly. This gives a performance boost to
> distributed clients that are aware of locality.
>
> Is my understanding wrong OR it doesn't apply to my scenario?
>
> Meanwhile I will try setting the parameter suggested by Lars and post you
> the results.
>
> Thanks,
> Ramu
>
>
> On Wed, Oct 9, 2013 at 2:29 PM, lars hofhansl  wrote:
>
>> Good call.
>> Could try to enable hbase.regionserver.checksum.verify, which will cause
>> HBase to do its own checksums rather than relying on HDFS (and which saves
>> 1 IO per block get).
>>
>> I do think you can expect the index blocks to be cached at all times.
>>
>> -- Lars
>> 
>> From: Vladimir Rodionov 
>> To: "user@hbase.apache.org" 
>> Sent: Tuesday, October 8, 2013 8:44 PM
>> Subject: RE: HBase Random Read latency > 100ms
>>
>>
>> Upd.
>>
>> Each HBase Get = 2 HDFS read IO (index block + data block)= 4 File IO
>> (data + .crc) in a worst case. I think if Bloom Filter is enabled than
>> it is going to be 6 File IO in a worst case (large data set), therefore
>> you will have not 5 IO requests in queue but up to 20-30 IO requests in a
>> queue
>> This definitely explains > 100ms avg latency.
>>
>>
>>
>> Best regards,
>> Vladimir Rodionov
>> Principal Platform Engineer
>> Carrier IQ, www.carrieriq.com
>> e-mail: vrodio...@carrieriq.com
>>
>> 
>>
>> From: Vladimir Rodionov
>> Sent: Tuesday, October 08, 2013 7:24 PM
>> To: user@hbase.apache.org
>> Subject: RE: HBase Random Read latency > 100ms
>>
>> Ramu,
>>
>> You have 8 server boxes and 10 client. You have 40 requests in parallel -
>> 5 per RS/DN?
>>
>> You have 5 requests on random reads in a IO queue of your single RAID1.
>> With avg read latency of 10 ms, 5 requests in queue will give us 30ms. Add
>> some overhead
>> of HDFS + HBase and you will probably have your issue explained ?
>>
>> Your bottleneck is your disk system, I think. When you serve most of
>> requests from disks as in your large data set scenario, make sure you have
>> adequate disk sub-system and
>> that it is configured properly. Block Cache and OS page can not help you
>> in this case as working data set is larger than both caches.
>>
>> Good performance numbers in small data set scenario are explained by the
>> fact that data fits into OS page cache and Block Cache - you do not read
>> data from disk even if
>> you disable block cache.
>>
>>
>> Best regards,
>> Vladimir Rodionov
>> Principal Platform Engineer
>> Carrier IQ, www.carrieriq.com
>> e-mail: vrodio...@carrieriq.com
>>
>> 
>> From: Ramu M S [ramu.ma...@gmail.com]
>> Sent: Tuesday, October 08, 2013 6:00 PM
>> To: user@hbase.apache.org
>> Subject: Re: HBase Random Read latency > 100ms
>>
>> Hi All,
>>
>> After few suggestions from the mails earlier I changed the following,
>>
>> 1. Heap Size to 16 GB
>> 2. Block Size to 16KB
>> 3. HFile size to 8 GB (Table now has 256 regions, 32 per server)

RE: Coprocessor Increments

[Vladimir Rodionov]

>>Contrived Example

>>Insert rowkey "California-12345" triggers a coprocessor to call
>>incrementColumnValue() with a rowkey of "California-total"  all on Region 1.

>>This would likely be on an insert on the same region. But as the table
>>grows, this secondary insert could end up on another region. If it is
>>confined, then suppose we later insert "California-95424" which still
>>triggers a call to incrementColumnValue() with a rowkey of
>>"California-total" all on Region 2.

>>Are we now left with two rowkeys of "California-total"? One on each
>>region server? If so, what happens if these two regions are compacted
>>into one?

>>Best regards,

Nope, Your "California-total" will migrate to Region 2 after region split is 
complete.

Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com



Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: Coprocessor Increments

[Vladimir Rodionov]

Classical deadlock :

CP-Region1 updates counter in CP-Region2 (waits on RPC)
CP-Region2 updates counter in CP-Region1 (waits on RPC)

I think its an anti-pattern. Do not do cross region calls in region CP code.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Michael Segel [msegel_had...@hotmail.com]
Sent: Thursday, October 10, 2013 9:55 AM
To: user@hbase.apache.org
Cc: Ted Yu
Subject: Re: Coprocessor Increments

I think Andrew has a handle on it… my take is that you end up running out of 
resources to handle an RPC connection while within your coprocessor and you're 
waiting for a resource to be free and it can't because another coprocessor has 
an RPC resource and is also waiting for a free resource.

Maybe its an over simplification, but if that's the case… you could always try 
thing to limit the RPC call, which would delay updating the counter. (Which may 
not be a problem) or redesign the coprocessors so that the coprocessors don't 
share the same RPC resources.

But the key is to really understanding and confirming what's causing the 
Deadlock in detail.

On Oct 10, 2013, at 11:15 AM, John Weatherford  
wrote:

> Michael,
> I would also really like to know how this issue is caused also. I can't even 
> give a solid way to reproduce our deadlock. It *seems* to happen more under 
> load, but nothing can be proved yet. While google-ing and looking for an 
> answer I came across that old message post  
> (http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5cn3ekzdjz_z-2qt8xo...@mail.gmail.com%3E).
>  This seemed to line up with what we are doing, so we _hope_ this will be a 
> fix for us, but we aren't entirely sure.
>
>
>
> On Thu 10 Oct 2013 07:57:46 AM PDT, Michael Segel wrote:
>> Can we just take a quick pause…
>>
>> John you wrote the following:
>> "We have been running into an RPC deadlock issue on HBase and from
>> investigation, we believe the root of the issue is in us doing cross
>> region increments from a coprocessor. After some further searching and
>> reading over this
>> <http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5cn3ekzdjz_z-2qt8xo...@mail.gmail.com%3E>
>> we think that we can solve this by doing the increments locally on the
>> region. "
>>
>> Which goes back to some thing Andrew wrote concerning an RPC deadlock.
>>
>> Can either you or Andrew explain in detail what is meant by the RPC
>> deadlock?
>>
>> This goes back to rethink how to implement coprocessors.
>>
>>
>> On Oct 9, 2013, at 11:03 PM, John Weatherford
>> mailto:john.weatherf...@telescope.tv>>
>> wrote:
>>
>>> Thank you ted. I might have to rethink my key design to accomodate
>>> this. With this design and using the totals in keys as you suggested,
>>> I would have to scan the entire "California" group to find the
>>> totals. Thank you very much for your help.
>>>
>>> -John
>>>
>>> On Wed 09 Oct 2013 08:43:12 PM PDT, Ted Yu wrote:
>>>> John:
>>>> Suppose 'California-**12346' is the start key of region 1 and
>>>> 'California-**
>>>> 95424' is the start key of region 2. You can choose
>>>> 'California-**12346#total'
>>>> to be the row key where increment is done for region 1 and
>>>> 'California-**95424#total'
>>>> to be the row key where increment is done for region 2.
>>>>
>>>> w.r.t. verification of whether given row key is indeed inside the
>>>> hosting
>>>> region, see the following:
>>>>
>>>> void checkRow(final byte [] row, String op) throws IOException {
>>>>
>>>>  if (!rowIsInRange(getRegionInfo(), row)) {
>>>>
>>>>throw new WrongRegionException("Requested row out of range for " +
>>>>
>>>>op + " on HRegion " + this + ", startKey='" +
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Wed, Oct 9, 2013 at 8:26 PM, John Weatherford <
>>>> john.weatherf...@telescope.tv
>>>> <mailto:john.weatherf...@telescope.tv>> wrote:
>>>>
>>>>> First, Thank you everyone for the quick response.
>>>>>
>>>>> Ted:
>>>>> The custom split policy could be an interesting solution.
>>>>> Regarding the
>>>>> other optio

RE: Coprocessor Increments

[Vladimir Rodionov]

Nope. It is not so obvious, but definitely the anti-pattern is still there. 

Each RPC call is served by a thread from RPC-handlers pool (which is 10? by 
default).

Making RPC call from within handler's therad is:

A. expensive
B. may result in some deadlock -type of situations when no more incoming RPC 
calls can be accepted because
   everyone is connected to everyone in a spaghetti way and waiting on RPC 
calls to complete

Let us say we have 2 region servers for simplicity:
RS1 and RS2. Both have pool of handler threads = 10

What happen when all 10 handlers in RS1  are trying to RPC RS2 and all 10 
handlers are trying to RPC RS2?

The same deadlock. Its all probabilistic .

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Vladimir Rodionov
Sent: Thursday, October 10, 2013 12:09 PM
To: user@hbase.apache.org
Subject: RE: Coprocessor Increments

Classical deadlock :

CP-Region1 updates counter in CP-Region2 (waits on RPC)
CP-Region2 updates counter in CP-Region1 (waits on RPC)

I think its an anti-pattern. Do not do cross region calls in region CP code.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Michael Segel [msegel_had...@hotmail.com]
Sent: Thursday, October 10, 2013 9:55 AM
To: user@hbase.apache.org
Cc: Ted Yu
Subject: Re: Coprocessor Increments

I think Andrew has a handle on it… my take is that you end up running out of 
resources to handle an RPC connection while within your coprocessor and you're 
waiting for a resource to be free and it can't because another coprocessor has 
an RPC resource and is also waiting for a free resource.

Maybe its an over simplification, but if that's the case… you could always try 
thing to limit the RPC call, which would delay updating the counter. (Which may 
not be a problem) or redesign the coprocessors so that the coprocessors don't 
share the same RPC resources.

But the key is to really understanding and confirming what's causing the 
Deadlock in detail.

On Oct 10, 2013, at 11:15 AM, John Weatherford  
wrote:

> Michael,
> I would also really like to know how this issue is caused also. I can't even 
> give a solid way to reproduce our deadlock. It *seems* to happen more under 
> load, but nothing can be proved yet. While google-ing and looking for an 
> answer I came across that old message post  
> (http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5cn3ekzdjz_z-2qt8xo...@mail.gmail.com%3E).
>  This seemed to line up with what we are doing, so we _hope_ this will be a 
> fix for us, but we aren't entirely sure.
>
>
>
> On Thu 10 Oct 2013 07:57:46 AM PDT, Michael Segel wrote:
>> Can we just take a quick pause…
>>
>> John you wrote the following:
>> "We have been running into an RPC deadlock issue on HBase and from
>> investigation, we believe the root of the issue is in us doing cross
>> region increments from a coprocessor. After some further searching and
>> reading over this
>> <http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5cn3ekzdjz_z-2qt8xo...@mail.gmail.com%3E>
>> we think that we can solve this by doing the increments locally on the
>> region. "
>>
>> Which goes back to some thing Andrew wrote concerning an RPC deadlock.
>>
>> Can either you or Andrew explain in detail what is meant by the RPC
>> deadlock?
>>
>> This goes back to rethink how to implement coprocessors.
>>
>>
>> On Oct 9, 2013, at 11:03 PM, John Weatherford
>> mailto:john.weatherf...@telescope.tv>>
>> wrote:
>>
>>> Thank you ted. I might have to rethink my key design to accomodate
>>> this. With this design and using the totals in keys as you suggested,
>>> I would have to scan the entire "California" group to find the
>>> totals. Thank you very much for your help.
>>>
>>> -John
>>>
>>> On Wed 09 Oct 2013 08:43:12 PM PDT, Ted Yu wrote:
>>>> John:
>>>> Suppose 'California-**12346' is the start key of region 1 and
>>>> 'California-**
>>>> 95424' is the start key of region 2. You can choose
>>>> 'California-**12346#total'
>>>> to be the row key where increment is done for region 1 and
>>>> 'California-**95424#total'
>>>> to be the row key where increment is done for region 2.
>>>>
>>>> w.r.t. verification of whether given row key is indeed inside the
>>>> hosting

RE: Coprocessor Increments

[Vladimir Rodionov]

Michael, 

RS has 3 different pools of threads (Readers (accept RPC), Handlers (process 
requests), third pool (do not remember name) of threads send request results 
back  to callers )
All of them communicate to through internal Queues. Cumbersome and needs to be  
refactored, of course.

All these pools are configurable (max threads). Readers and writers are fine - 
they will never get into any deadlock situation.
Handlers are different. In my example, I described the situation when ALL 
handlers in RS1 are waiting on RPC calls to RS2 and
ALL handlers in RS2 are waiting on RPC calls to RS2. For the sake of simplicity:

RS1 and RS2 - two region servers
handlers count = 1 - just one thread in a pool.

1. RS2 coprocessor sends RPC to RS1. From single handler thread. Total 
available handlers in RS2 = 0.  
2. RS1 receives request from RS2 CP2 (coprocessor). 
3. RS1 handler1 receives request. Total available handlers in RS1 = 0;
4. RS1 coprocessor makes RPC call back to RS2
5. RS2 Reader thread places request into Handler pool queue, but there is no 
handlers available (see 1). Deadlock.
and the request MUST timeout (fail) eventually. If it does not fail - there is 
the issue in HBase RPC.

But technically, its a deadlock.



Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Michael Segel [msegel_had...@hotmail.com]
Sent: Thursday, October 10, 2013 2:57 PM
To: user@hbase.apache.org
Subject: Re: Coprocessor Increments

Ok…

So…  I see you followed up to your earlier post.

Lets walk through this to make sure we're on the same page.

You put()  row 1 in table 1.
The post_put() wants to insert a value in table 2.

Now suppose I have an htable pool of connections in my Region Observer.
(Is this overkill? I'm treating the RO as if its a client connecting to HBase.)

RO performs a put() into the second table.

The RPC handlers are a Region server resource, yes?

So I can always increase them from the default (10) … but the point is that I 
can have 10 clients updating table A and then have a couple of different 
regions on the RS for the table making RO requests.

Is that the case?


On Oct 10, 2013, at 2:23 PM, Vladimir Rodionov  wrote:

> Nope. It is not so obvious, but definitely the anti-pattern is still there.
>
> Each RPC call is served by a thread from RPC-handlers pool (which is 10? by 
> default).
>
> Making RPC call from within handler's therad is:
>
> A. expensive
> B. may result in some deadlock -type of situations when no more incoming RPC 
> calls can be accepted because
> everyone is connected to everyone in a spaghetti way and waiting on RPC calls 
> to complete
>
> Let us say we have 2 region servers for simplicity:
> RS1 and RS2. Both have pool of handler threads = 10
>
> What happen when all 10 handlers in RS1  are trying to RPC RS2 and all 10 
> handlers are trying to RPC RS2?
>
> The same deadlock. Its all probabilistic .
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodio...@carrieriq.com
>
> 
> From: Vladimir Rodionov
> Sent: Thursday, October 10, 2013 12:09 PM
> To: user@hbase.apache.org
> Subject: RE: Coprocessor Increments
>
> Classical deadlock :
>
> CP-Region1 updates counter in CP-Region2 (waits on RPC)
> CP-Region2 updates counter in CP-Region1 (waits on RPC)
>
> I think its an anti-pattern. Do not do cross region calls in region CP code.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodio...@carrieriq.com
>
> 
> From: Michael Segel [msegel_had...@hotmail.com]
> Sent: Thursday, October 10, 2013 9:55 AM
> To: user@hbase.apache.org
> Cc: Ted Yu
> Subject: Re: Coprocessor Increments
>
> I think Andrew has a handle on it… my take is that you end up running out of 
> resources to handle an RPC connection while within your coprocessor and 
> you're waiting for a resource to be free and it can't because another 
> coprocessor has an RPC resource and is also waiting for a free resource.
>
> Maybe its an over simplification, but if that's the case… you could always 
> try thing to limit the RPC call, which would delay updating the counter. 
> (Which may not be a problem) or redesign the coprocessors so that the 
> coprocessors don't share the same RPC resources.
>
> But the key is to really understanding and confirming what's causing the 
> Deadlock in detail.
>
> On Oct 10, 2013, at 11:15 AM, John Weatherford 
>  wrote:
>
>> Michael,
>> I would also really like to know how this issue is caused also. I can't ev

RE: Coprocessor Increments

[Vladimir Rodionov]

>>With respect to the OP's design… does the deadlock occur because he's trying 
>>to update a column in a different row within the same table?

Because he is trying to update *row* in a different Region (and potentially in 
different RS).

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Michael Segel [msegel_had...@hotmail.com]
Sent: Friday, October 11, 2013 9:10 AM
To: user@hbase.apache.org
Cc: Vladimir Rodionov
Subject: Re: Coprocessor Increments


Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: Limiting the number of regions per RS

[Vladimir Rodionov]

You can write your own balancer and disable standard one:
hbase.balancer.period = LONG.MAX_VALUE

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: e...@gigya-inc.com [e...@gigya-inc.com] On Behalf Of Eran Kutner 
[e...@gigya.com]
Sent: Monday, October 14, 2013 12:15 PM
To: user@hbase.apache.org
Subject: Limiting the number of regions per RS

Hi,
We have a cluster with unequal machines, some are newer with more disk
space, more RAM and stronger CPUs. HDFS amd M/R can be tuned to consume
less resources but I was unable to find a way to cause HBase balancer to
put less regions on the weaker servers, so now they have the same number of
regions as the strong servers, causing them to access HDFS remotely because
they don't have enough local disk space to store all their region files.

I noticed that in version 0.97 the stochastic load balancer has some
configuration options that might help, but it seems that it doesn't exist
in version 0.94.6, which is what I'm using (based on CDH4.3.1).

Any suggestions?


Thanks.

-eran

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: HBase load problems

[Vladimir Rodionov]

There is a pressure on memstore to flush more frequently and create smaller 
store files when you have too many active regions (50 in your case)
if you have all your settings default you allocate only 40% of heap to  
memstores. If you have say 8GB heap - memstore has 3.2GB or
You flush size is going to be 3.2/(50* column family number) < 60MB. You create 
60 MB store files and you need to run compaction more frequently
in this case.

What options do you have:

1. Increase heap (if not large already) and/or increase memstore size from 
default 0.4 (do not forget to decrease block cache)
2. Increase region size
3. You may play with WAL log size (default is HDFS block size) and with number 
of WAL files per Region. Both needs to be increased as well.

Monitor split/compaction activity (check HBase book how to do this)

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Kireet [kir...@feedly.com]
Sent: Wednesday, October 16, 2013 8:47 AM
To: user@hbase.apache.org
Subject: HBase load problems

Over the past couple of months we have seen a significant increase in
datanode I/O load in our cluster, an increase of 100% in disk read/write
rates while our application requests have increased by a much smaller
amount, perhaps 5-10%. The read/write rate has been increasing gradually
over time.

The data size of our cluster has increased quite a bit. In particular we
have one table that is keyed by randomized timestamp (random bytes +
timestamp). It has grown at about 40GB/day (before replication) with an
average row size of about 1KB in a single column. It makes up about 80%
of our total data size and is at about 50 regions per data node. Our
first guess is the issue has something to do with this table since it
dominates the cluster data size.

We are considering splitting the table into multiple tables organized by
timestamp. 90% or more of reads/writes are for recent data, so our
thinking is we could keep the "most recent data" table much smaller by
doing this and perhaps make it easier for hbase to optimize things.
E.g., compactions would be quicker and perhaps the block cache would
become more effective as each block would have recent data instead of a
continually decreasing fraction.

However, this would be a big code change and we would like to confirm
as much as possible that this is the true problem. What are the key
metrics we should look at for confirmation?

Also we don't have short circuit reads enabled at the moment. We have
seen articles on the web claiming big improvements in some cases but no
change in others. Are there particular characteristics of systems that
will see big improvements when this setting is enabled?


Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: HBase load problems

[Vladimir Rodionov]

afaik,  your keys are already:
salted_prefix + timestamp

If cardinality of salted_prefix is not than big and it should not be (we use - 
256 in our system) you will have already your rows grouped by
time in HFile blocks, I think you should have pretty pretty good locality of 
data in a block cache.
What is the cardinality of salted_prefix? And how many rows to  you ingest per 
day?

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Kireet [kir...@feedly.com]
Sent: Thursday, October 17, 2013 5:56 AM
To: user@hbase.apache.org
Subject: Re: HBase load problems

Sorry for the confusion, I was trying to get at if it would be a large
operation or there would be a large disruption right after increasing
the max region size. It sounds like it wouldn't.

Getting back to the original question, if we do 90%+ of our reads/writes
to very recent data, would it make sense to keep that in a separate
table? It seems like that may keep things more optimized, the block
cache would be more efficient, compactions would run quicker with much
less data on the 'recent data' table and perhaps less often on the
'older data' table(s), etc.

On 10/17/13 12:52 AM, Stack wrote:
> On Wed, Oct 16, 2013 at 6:26 PM, Kireet  wrote:
>
>> is there a downside to going to larger regions?
>
>
>
> Generally we see pluses (See 2.5.6.2 Bigger Regions in
> http://hbase.apache.org/book/important_configurations.html for the latest
> scripture on the topic).  Downsides would be something like compactions run
> longer (coarsely, same overall work, just takes longer to complete a region)
>
>
>
>> It looks like merge is a larger operation, so changing the config setting
>> would simply cause existing regions to grow to the larger value right?
>> Would the way to confirm this be to check compaction rates and also maybe
>> the size of store files in hdfs?
>>
>>
> I'm not sure I understand the question above Kireet.  You doing merges?
>   Yes, you can just change the configs on the cluster and regions will just
> grow (will need to rolling restart the cluster to pick up the config.)
>
> Please ask again what you would confirm.
>
> Thanks,
> St.Ack
> P.S. Feedly-fan here.
>


Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: HBase load problems

[Vladimir Rodionov]

Sure, if you can do this - do this. Its the only way to reduce compaction 
significantly
but downside is increased complexity in data store maintenance, but its usual:
something to loose - something to gain.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Kireet [kir...@feedly.com]
Sent: Thursday, October 17, 2013 11:21 AM
To: user@hbase.apache.org
Subject: Re: HBase load problems

Perhaps this is the problem, our salt is 4 bytes, which completely
randomizes things. We create about 35m rows/day. What we are thinking is
that if older data isn't changed, then if we split recent data into it's
own table then compactions would be much cheaper because the older data
wouldn't need to be rewritten.

On 10/17/13 1:46 PM, Vladimir Rodionov wrote:
> afaik,  your keys are already:
> salted_prefix + timestamp
>
> If cardinality of salted_prefix is not than big and it should not be (we use 
> - 256 in our system) you will have already your rows grouped by
> time in HFile blocks, I think you should have pretty pretty good locality of 
> data in a block cache.
> What is the cardinality of salted_prefix? And how many rows to  you ingest 
> per day?
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodio...@carrieriq.com
>
> 
> From: Kireet [kir...@feedly.com]
> Sent: Thursday, October 17, 2013 5:56 AM
> To: user@hbase.apache.org
> Subject: Re: HBase load problems
>
> Sorry for the confusion, I was trying to get at if it would be a large
> operation or there would be a large disruption right after increasing
> the max region size. It sounds like it wouldn't.
>
> Getting back to the original question, if we do 90%+ of our reads/writes
> to very recent data, would it make sense to keep that in a separate
> table? It seems like that may keep things more optimized, the block
> cache would be more efficient, compactions would run quicker with much
> less data on the 'recent data' table and perhaps less often on the
> 'older data' table(s), etc.
>
> On 10/17/13 12:52 AM, Stack wrote:
>> On Wed, Oct 16, 2013 at 6:26 PM, Kireet  wrote:
>>
>>> is there a downside to going to larger regions?
>>
>>
>>
>> Generally we see pluses (See 2.5.6.2 Bigger Regions in
>> http://hbase.apache.org/book/important_configurations.html for the latest
>> scripture on the topic).  Downsides would be something like compactions run
>> longer (coarsely, same overall work, just takes longer to complete a region)
>>
>>
>>
>>> It looks like merge is a larger operation, so changing the config setting
>>> would simply cause existing regions to grow to the larger value right?
>>> Would the way to confirm this be to check compaction rates and also maybe
>>> the size of store files in hdfs?
>>>
>>>
>> I'm not sure I understand the question above Kireet.  You doing merges?
>>Yes, you can just change the configs on the cluster and regions will just
>> grow (will need to rolling restart the cluster to pick up the config.)
>>
>> Please ask again what you would confirm.
>>
>> Thanks,
>> St.Ack
>> P.S. Feedly-fan here.
>>
>
>
> Confidentiality Notice:  The information contained in this message, including 
> any attachments hereto, may be confidential and is intended to be read only 
> by the individual or entity to whom this message is addressed. If the reader 
> of this message is not the intended recipient or an agent or designee of the 
> intended recipient, please note that any review, use, disclosure or 
> distribution of this message or its attachments, in any form, is strictly 
> prohibited.  If you have received this message in error, please immediately 
> notify the sender and/or notificati...@carrieriq.com and delete or destroy 
> any copy of this message and its attachments.
>

RE: CDH4.4 and HBASE-8912 issue

[Vladimir Rodionov]

Boris, what does hbck say?

We have had this issue a couple times before. To fix it I had to stop the 
cluster, run offline meta repair tool, delete zk-store on each zk quorum node
Offline Meta repair tool will not work if there are  inconsistencies  in HBase 
- you better try hbase hbck -fixAll first.


Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: lars hofhansl [la...@apache.org]
Sent: Thursday, October 17, 2013 10:47 AM
To: user@hbase.apache.org; Harsh J
Subject: Re: CDH4.4 and HBASE-8912 issue

Hi Boris,

could you add the exact exceptions/logs you've seen to HBASE-8912?

Thanks.

-- Lars




 From: Boris Emelyanov 
To: Harsh J ; "user@hbase.apache.org" 

Sent: Thursday, October 17, 2013 3:54 AM
Subject: Re: CDH4.4 and HBASE-8912 issue


On 17.10.2013 13:43, Harsh J wrote:
> Moving tou...@hbase.apache.org.
>
> HBASE-8912 is still unresolved, and 0.94.13 is just the targeted
> version presently.
>
> Are you certain this is the exact issue you're hitting? I believe you
> can workaround this by removing the specific bad znode in ZK or so.
> When starting up after your major upgrade, did you ensure cleaning
> your ZK /hbase znode?
>
> On Thu, Oct 17, 2013 at 2:55 PM, Boris Emelyanov  wrote:
>> Hello! I've just upgraded my hadoop test-cluster from cdh3 to cdh4.4 with
>> hbase-0.94.6 and have facedhttps://issues.apache.org/jira/browse/HBASE-8912
>> issue.
>>
>> Suggested solution was to update hbase to version 0.94.13, which is absent
>> in cloudera distribution.
>>
>> Is it possible to run pure hbase over cloudera hadoop?
>>
>> Or how can i find if this bug is present in previous versions of cdh?
>>
>> --
>> Best regards,
>>
>> Boris Emelyanov.
>

> Are you certain this is the exact issue you're hitting?

I gues so. Description suits just fine!

> I believe you can workaround this by removing the specific bad znode
in ZK or so.

When I disable loadbalancer just after master startup (balance_switch
false), cluster works fine.
But when loadbalancer is enabled and it starts, it allways fails with
exception described in issue, allways with different regions.

>When starting up after your major upgrade, did you ensure cleaning
your ZK /hbase znode?

Yes, of course.


--
Best regards,

Boris Emelyanov.

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: Block cache hit ratio question

[Vladimir Rodionov]

HBase block cache does not expose any statistics on a per-table basis. This 
looks like a nice to have feature in
a next releases. If your load is mostly random reads, than several factors 
affect the overall performance:

1. The avg read size / block size ratio. The large ratio - the better block 
cache hit ration you will have.
2. Hot data set size.
3 Available RAM for block cache

For example,

Block size = 64K
Avg read size = 1K
Hot data set size = 1GB

You will need 64GB of block cache to cache 100% your random reads.
Usually, for random reads the smaller block size - the better, but you can not 
decrease block size w/o additional overhead of bookkeeping
larger number of smaller blocks. Your actual limit is around 16KB.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Vaibhav Puranik [vpura...@gmail.com]
Sent: Friday, October 18, 2013 3:20 PM
To: user@hbase.apache.org
Subject: Block cache hit ratio question

Hi all,

Our HBase is slowing down because of lots of disk reads. We are trying to
analyze it. We have lots of real time random reads. And We found that HBase
is constantly reading from disk instead of from memory.

Looks like block cache hit ratio is a good place to start.

Is there any way we can get block cache hit ratio by table or by region? I
could only find it per region server in region server UI. I couldn't find
it in API either.

Feel free to suggest any other metrics we should be looking at too.

Thanks,
Vaibhav
GumGum

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: 'hbase.client.scanner.caching' default value for HBase 0.90.6?

[Vladimir Rodionov]

HBase bible (book) have answers on *almost* all questions (1GB default heap is 
too low for many applications - that is why you see OOME).
HBase book:
http://hbase.apache.org/book.html

+1 on version upgrade.

Is there any particular reason for so old version of HBase? The current is 
0.94.12.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: A Laxmi [a.lakshmi...@gmail.com]
Sent: Sunday, October 20, 2013 6:01 PM
To: user@hbase.apache.org
Subject: Re: 'hbase.client.scanner.caching' default value for HBase 0.90.6?

Thanks Jean! But the problem is I am using a crawler "Apache Nutch" that
supports a HBase version 0.90.6 so I have no option than to restrict to
0.90.6 version. :(

I am also getting an out of memory error: heap space while using HBase. I
am using default 1GB heap setting in hbase-en.sh. I have 6GB RAM. Please
advise.

 I have set  -

*hbase.client.scanner.caching* to 1. *
hbase.client.scanner.timeout.period *to* 12
hbase.regionserver.handler.count* to 10. *
hbase.rpc.timeout *to 90
*hbase.regionserver.lease.period* to 90
*hbase.client.write.buffer* to 2097152

Please advise what can be done to avoid out of memory error: heap space.



On Sun, Oct 20, 2013 at 8:49 PM, Jean-Marc Spaggiari <
jean-m...@spaggiari.org> wrote:

> Also, Laxmi, if possible, I will highly recommend you to migrate to a more
> recent version. 0.90 is VERY old...
>
>
> 2013/10/20 lars hofhansl 
>
> > In any version < 0.96 it is 1. So in 0.90.6 it would be 1.
> >
> > -- Lars
> >
> >
> >
> > - Original Message -
> > From: A Laxmi 
> > To: user@hbase.apache.org
> > Cc:
> > Sent: Sunday, October 20, 2013 11:17 AM
> > Subject: 'hbase.client.scanner.caching' default value for HBase 0.90.6?
> >
> > Hi,
> >
> > What is the default value of '*hbase.client.scanner.caching*' for *HBase
> > 0.90.6* version in particular? Is it 1 or 100?
> >
> > Thanks
> >
> >
>

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: row filter - binary comparator at certain range

[Vladimir Rodionov]

FuzzyRowFilter does not work on sub-key ranges.
Salting is bad for any scan operation, unfortunately. When salt prefix 
cardinality is small (1-2 bytes),
one can try something similar to FuzzyRowFilter but with additional sub-key 
range support.
If salt prefix cardinality is high (> 2 bytes) - do a full scan with your own 
Filter (for timestamp ranges).

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Premal Shah [premal.j.s...@gmail.com]
Sent: Sunday, October 20, 2013 10:42 PM
To: user
Subject: Re: row filter - binary comparator at certain range

Have you looked at FuzzyRowFilter? Seems to me that it might satisfy your
use-case.
http://blog.sematext.com/2012/08/09/consider-using-fuzzyrowfilter-when-in-need-for-secondary-indexes-in-hbase/


On Sun, Oct 20, 2013 at 9:31 PM, Tony Duan  wrote:

> Alex Vasilenko  writes:
>
> >
> > Lars,
> >
> > But how it will behave, when I have salt at the beginning of the key to
> > properly shard table across regions? Imagine row key of format
> > salt:timestamp and rows goes like this:
> > ...
> > 1:15
> > 1:16
> > 1:17
> > 1:23
> > 2:3
> > 2:5
> > 2:12
> > 2:15
> > 2:19
> > 2:25
> > ...
> >
> > And I want to find all rows, that has second part (timestamp) in range
> > 15-25. What startKey and endKey should be used?
> >
> > Alexandr Vasilenko
> > Web Developer
> > Skype:menterr
> > mob: +38097-611-45-99
> >
> > 2012/2/9 lars hofhansl 
> Hi,
> Alexandr Vasilenko
> Have you ever resolved this issue?i am also facing this iusse.
> i also want implement this functionality.
> Imagine row key of format
> salt:timestamp and rows goes like this:
> ...
> 1:15
> 1:16
> 1:17
> 1:23
> 2:3
> 2:5
> 2:12
> 2:15
> 2:19
> 2:25
> ...
>
> And I want to find all rows, that has second part (timestamp) in range
> 15-25.
>
> Could you please tell me how you resolve this ?
> thanks  in advance.
>
>
> Tony duan
>
>


--
Regards,
Premal Shah.

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: row filter - binary comparator at certain range

[Vladimir Rodionov]

I advise you to refactor your key.

1. First, use salting of a low cardinality (say 1 random byte)
2. To improve range query - add time bucket to your time dimensions:

KEY: 
salt_timebucket_time

tiimebucket is something similar to: day_hour_min
time   - sec+ms part of timestamp

It will be easier for you to rollup events by tiimebucket or part of it:
byte day, by hour, by min, but you will need your own filter to efficiently 
implement functionality (range query).
FuzzyRowFilter will work for small ranges, but for large ranges performance 
will degrade significantly.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Vladimir Rodionov
Sent: Monday, October 21, 2013 9:14 AM
To: user@hbase.apache.org
Subject: RE: row filter - binary comparator at certain range

FuzzyRowFilter does not work on sub-key ranges.
Salting is bad for any scan operation, unfortunately. When salt prefix 
cardinality is small (1-2 bytes),
one can try something similar to FuzzyRowFilter but with additional sub-key 
range support.
If salt prefix cardinality is high (> 2 bytes) - do a full scan with your own 
Filter (for timestamp ranges).

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Premal Shah [premal.j.s...@gmail.com]
Sent: Sunday, October 20, 2013 10:42 PM
To: user
Subject: Re: row filter - binary comparator at certain range

Have you looked at FuzzyRowFilter? Seems to me that it might satisfy your
use-case.
http://blog.sematext.com/2012/08/09/consider-using-fuzzyrowfilter-when-in-need-for-secondary-indexes-in-hbase/


On Sun, Oct 20, 2013 at 9:31 PM, Tony Duan  wrote:

> Alex Vasilenko  writes:
>
> >
> > Lars,
> >
> > But how it will behave, when I have salt at the beginning of the key to
> > properly shard table across regions? Imagine row key of format
> > salt:timestamp and rows goes like this:
> > ...
> > 1:15
> > 1:16
> > 1:17
> > 1:23
> > 2:3
> > 2:5
> > 2:12
> > 2:15
> > 2:19
> > 2:25
> > ...
> >
> > And I want to find all rows, that has second part (timestamp) in range
> > 15-25. What startKey and endKey should be used?
> >
> > Alexandr Vasilenko
> > Web Developer
> > Skype:menterr
> > mob: +38097-611-45-99
> >
> > 2012/2/9 lars hofhansl 
> Hi,
> Alexandr Vasilenko
> Have you ever resolved this issue?i am also facing this iusse.
> i also want implement this functionality.
> Imagine row key of format
> salt:timestamp and rows goes like this:
> ...
> 1:15
> 1:16
> 1:17
> 1:23
> 2:3
> 2:5
> 2:12
> 2:15
> 2:19
> 2:25
> ...
>
> And I want to find all rows, that has second part (timestamp) in range
> 15-25.
>
> Could you please tell me how you resolve this ?
> thanks  in advance.
>
>
> Tony duan
>
>


--
Regards,
Premal Shah.

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: row filter - binary comparator at certain range

[Vladimir Rodionov]

James, good to hear that Phoenix supports these features.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: James Taylor [jtay...@salesforce.com]
Sent: Monday, October 21, 2013 9:37 AM
To: user@hbase.apache.org
Subject: Re: row filter - binary comparator at certain range

Phoenix restricts salting to a single byte.
Salting perhaps is misnamed, as the salt byte is a stable hash based on the
row key.
Phoenix's skip scan supports sub-key ranges.
We've found salting in general to be faster (though there are cases where
it's not), as it ensures better parallelization.

Regards,
James



On Mon, Oct 21, 2013 at 9:14 AM, Vladimir Rodionov
wrote:

> FuzzyRowFilter does not work on sub-key ranges.
> Salting is bad for any scan operation, unfortunately. When salt prefix
> cardinality is small (1-2 bytes),
> one can try something similar to FuzzyRowFilter but with additional
> sub-key range support.
> If salt prefix cardinality is high (> 2 bytes) - do a full scan with your
> own Filter (for timestamp ranges).
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodio...@carrieriq.com
>
> 
> From: Premal Shah [premal.j.s...@gmail.com]
> Sent: Sunday, October 20, 2013 10:42 PM
> To: user
> Subject: Re: row filter - binary comparator at certain range
>
> Have you looked at FuzzyRowFilter? Seems to me that it might satisfy your
> use-case.
>
> http://blog.sematext.com/2012/08/09/consider-using-fuzzyrowfilter-when-in-need-for-secondary-indexes-in-hbase/
>
>
> On Sun, Oct 20, 2013 at 9:31 PM, Tony Duan  wrote:
>
> > Alex Vasilenko  writes:
> >
> > >
> > > Lars,
> > >
> > > But how it will behave, when I have salt at the beginning of the key to
> > > properly shard table across regions? Imagine row key of format
> > > salt:timestamp and rows goes like this:
> > > ...
> > > 1:15
> > > 1:16
> > > 1:17
> > > 1:23
> > > 2:3
> > > 2:5
> > > 2:12
> > > 2:15
> > > 2:19
> > > 2:25
> > > ...
> > >
> > > And I want to find all rows, that has second part (timestamp) in range
> > > 15-25. What startKey and endKey should be used?
> > >
> > > Alexandr Vasilenko
> > > Web Developer
> > > Skype:menterr
> > > mob: +38097-611-45-99
> > >
> > > 2012/2/9 lars hofhansl 
> > Hi,
> > Alexandr Vasilenko
> > Have you ever resolved this issue?i am also facing this iusse.
> > i also want implement this functionality.
> > Imagine row key of format
> > salt:timestamp and rows goes like this:
> > ...
> > 1:15
> > 1:16
> > 1:17
> > 1:23
> > 2:3
> > 2:5
> > 2:12
> > 2:15
> > 2:19
> > 2:25
> > ...
> >
> > And I want to find all rows, that has second part (timestamp) in range
> > 15-25.
> >
> > Could you please tell me how you resolve this ?
> > thanks  in advance.
> >
> >
> > Tony duan
> >
> >
>
>
> --
> Regards,
> Premal Shah.
>
> Confidentiality Notice:  The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited.  If you have received this message in error, please
> immediately notify the sender and/or notificati...@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: What cause region server to timeout other than long gc?

[Vladimir Rodionov]

VM swapping? Did you check swapping?

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Henry Hung [ythu...@winbond.com]
Sent: Tuesday, October 22, 2013 8:41 PM
To: user@hbase.apache.org
Subject: What cause region server to timeout other than long gc?

Hi All,

Today I have 1 of 9 region servers down because of zookeeper timeout, this is 
the log:
2013-10-23 07:41:34,139 INFO org.apache.hadoop.hbase.regionserver.Store: 
Starting compaction of 4 file(s) in cf of 
MES_PERF_LOG_TIME,\x00\x04\x00\x00\x01A\x9D\xDD\xD9\x8D\x00\x00\x08\xD0fcap2\x00\x00=\xFC,1381922811424.cc6325e7896e3844d124321c315b02b0.
 into 
tmpdir=hdfs://fchddn1.ctfab.com:9000/hbase/MES_PERF_LOG_TIME/cc6325e7896e3844d124321c315b02b0/.tmp,
 seqid=2655801231, totalSize=923.8m
2013-10-23 07:41:34,139 DEBUG org.apache.hadoop.hbase.regionserver.Compactor: 
Compacting 
hdfs://fchddn1.ctfab.com:9000/hbase/MES_PERF_LOG_TIME/cc6325e7896e3844d124321c315b02b0/cf/3ae8a77f4a669b93eba2068f5778,
 keycount=34504344, bloomtype=NONE, size=501.3m, encoding=NONE
2013-10-23 07:41:34,139 DEBUG org.apache.hadoop.hbase.regionserver.Compactor: 
Compacting 
hdfs://fchddn1.ctfab.com:9000/hbase/MES_PERF_LOG_TIME/cc6325e7896e3844d124321c315b02b0/cf/055361f74fb644f1b3eb48f5da7e3ec0,
 keycount=18581704, bloomtype=NONE, size=269.5m, encoding=NONE
2013-10-23 07:41:34,139 DEBUG org.apache.hadoop.hbase.regionserver.Compactor: 
Compacting 
hdfs://fchddn1.ctfab.com:9000/hbase/MES_PERF_LOG_TIME/cc6325e7896e3844d124321c315b02b0/cf/a92cf779dca44843ab4c5130cca68d4f,
 keycount=9953404, bloomtype=NONE, size=143.9m, encoding=NONE
2013-10-23 07:41:34,139 DEBUG org.apache.hadoop.hbase.regionserver.Compactor: 
Compacting 
hdfs://fchddn1.ctfab.com:9000/hbase/MES_PERF_LOG_TIME/cc6325e7896e3844d124321c315b02b0/cf/51b89f361eb64f2f8c894a5c1516c604,
 keycount=664008, bloomtype=NONE, size=9.2m, encoding=NONE
2013-10-23 07:41:34,182 DEBUG org.apache.hadoop.hbase.util.FSUtils: Creating 
file=hdfs://fchddn1.ctfab.com:9000/hbase/MES_PERF_LOG_TIME/cc6325e7896e3844d124321c315b02b0/.tmp/16a28f8fc9f049c0a8b014502faf6339
 with permission=rwxrwxrwx
2013-10-23 07:41:34,190 DEBUG org.apache.hadoop.hbase.io.hfile.HFileWriterV2: 
Initialized with CacheConfig:enabled [cacheDataOnRead=true] 
[cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] 
[cacheEvictOnClose=false] [cacheCompressed=false]
2013-10-23 07:41:34,191 INFO org.apache.hadoop.hbase.regionserver.StoreFile: 
Delete Family Bloom filter type for 
hdfs://fchddn1.ctfab.com:9000/hbase/MES_PERF_LOG_TIME/cc6325e7896e3844d124321c315b02b0/.tmp/16a28f8fc9f049c0a8b014502faf6339:
 CompoundBloomFilterWriter
2013-10-23 07:43:16,793 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 
51848ms instead of 3000ms, this is likely due to a long garbage collecting 
pause and it's usually bad, see 
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-10-23 07:43:16,799 INFO org.apache.zookeeper.ClientCnxn: Client session 
timed out, have not heard from server in 60261ms for sessionid 
0x1412996eb5d2170, closing socket connection and attempting reconnect
2013-10-23 07:43:16,800 INFO org.apache.zookeeper.ClientCnxn: Client session 
timed out, have not heard from server in 60184ms for sessionid 
0x1412996eb5d217e, closing socket connection and attempting reconnect
2013-10-23 07:43:16,807 WARN org.apache.hadoop.ipc.HBaseServer: 
(responseTooSlow): 
{"processingtimems":50480,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@3b17c1aa),
 rpc version=1, client version=29, 
methodsFingerPrint=-1368823753","client":"10.16.10.181:40987","starttimems":1382485346321,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
2013-10-23 07:43:16,807 WARN org.apache.hadoop.ipc.HBaseServer: 
(responseTooSlow): 
{"processingtimems":50159,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@71f59d1a),
 rpc version=1, client version=29, 
methodsFingerPrint=-1368823753","client":"10.16.10.181:40023","starttimems":1382485346321,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
2013-10-23 07:43:16,808 WARN org.apache.hadoop.ipc.HBaseServer: 
(responseTooSlow): 
{"processingtimems":50480,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@380e6d06),
 rpc version=1, client version=29, 
methodsFingerPrint=-1368823753","client":"10.16.10.181:40963","starttimems":1382485346321,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
2013-10-23 07:43:18,154 INFO org.apac

RE: filtering rows for column values not null

[Vladimir Rodionov]

If 'firstName' is NULL - it is missing completely from a row. Add explicitly 
this column to Scan you create:

scan.addColumn(YOUR_CF, 'firstName'.getBytes());

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Ted [r6squee...@gmail.com]
Sent: Thursday, October 24, 2013 1:54 AM
To: user@hbase.apache.org
Subject: filtering rows for column values not null

Hi I have a relatively simple situation,

As an example I have a table of Users, with first and last name.

I set a scan a FilterList and I add a SingleColumnValueFilter, with
column qualifier=firstName, CompareOp.EQUAL, and value="bob".

The problem is, I'm getting "bob" as well as anyone with out a firstName.

I tried adding NOT_EQUALS null and NOT_EQUALS (new byte[0]) and neither helped.

Does anyone know of how to filter on a column value not null?
--
Ted.

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: filtering rows for column values not null

[Vladimir Rodionov]

This will work ONLY if you add single column to scan. If you scan multiple 
columns you will
need additional filter (reverse SkipFilter) which filter outs all rows (outputs 
of SingleColumnValueFilter) which
do not have 'firstName' column. I do not think HBase provide similar filter but 
you can create it yourself.

See SkipFilter java source

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com

____
From: Vladimir Rodionov
Sent: Thursday, October 24, 2013 9:07 AM
To: user@hbase.apache.org
Subject: RE: filtering rows for column values not null

If 'firstName' is NULL - it is missing completely from a row. Add explicitly 
this column to Scan you create:

scan.addColumn(YOUR_CF, 'firstName'.getBytes());

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Ted [r6squee...@gmail.com]
Sent: Thursday, October 24, 2013 1:54 AM
To: user@hbase.apache.org
Subject: filtering rows for column values not null

Hi I have a relatively simple situation,

As an example I have a table of Users, with first and last name.

I set a scan a FilterList and I add a SingleColumnValueFilter, with
column qualifier=firstName, CompareOp.EQUAL, and value="bob".

The problem is, I'm getting "bob" as well as anyone with out a firstName.

I tried adding NOT_EQUALS null and NOT_EQUALS (new byte[0]) and neither helped.

Does anyone know of how to filter on a column value not null?
--
Ted.

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: Add Columnsize Filter for Scan Operation

[Vladimir Rodionov]

Using HBase client API (scanners) for M/R is so oldish :). HFile has well 
defined format and it is much more efficient to read them directly.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Dhaval Shah [prince_mithi...@yahoo.co.in]
Sent: Thursday, October 24, 2013 9:53 AM
To: user@hbase.apache.org
Subject: Re: Add Columnsize Filter for Scan Operation

Jean, if we don't add setBatch to the scan, MR job does cause HBase to crash 
due to OOME. We have run into this in the past as well. Basically the problem 
is - Say I have a region server with 12GB of RAM and a row of size 20GB (an 
extreme example, in practice, HBase runs out of memory way before 20GB). If I 
query the entire row, HBase does not have enough memory to hold/process it for 
the response.

In practice, if your setCaching > 1, then the aggregate of all rows growing too 
big can also cause the same issue.

I think 1 way we can solve this issue is making the HBase server serve 
responses in a streaming fashion somehow (not exactly sure about the details on 
how this can work but if it has to hold the entire row in memory, its going to 
be bound by HBase heap size)

Regards,
Dhaval



 From: Jean-Marc Spaggiari 
To: user 
Sent: Thursday, 24 October 2013 12:37 PM
Subject: Re: Add Columnsize Filter for Scan Operation


If the MR crash because of the number of columns, then we have an issue
that we need to fix ;) Please open a JIRA provide details if you are facing
that.

Thanks,

JM



2013/10/24 John 

> @Jean-Marc: Sure, I can do that, but thats a little bit complicated because
> the the rows has sometimes Millions of Columns and I have to handle them
> into different batches because otherwise hbase crashs. Maybe I will try it
> later, but first I want to try the API version. It works okay so far, but I
> want to improve it a little bit.
>
> @Ted: I try to modify it, but I have no idea how exactly do this. I've to
> count the number of columns in that filter (that works obviously with the
> count field). But there is no Method that is caleld after iterating over
> all elements, so I can not return the Drop ReturnCode in the filterKeyValue
> Method because I did'nt know when it was the last one. Any ideas?
>
> regards
>
>
> 2013/10/24 Ted Yu 
>
> > Please take a look
> > at
> src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java :
> >
> >  * Simple filter that returns first N columns on row only.
> >
> > You can modify the filter to suit your needs.
> >
> > Cheers
> >
> >
> > On Thu, Oct 24, 2013 at 7:52 AM, John 
> wrote:
> >
> > > Hi,
> > >
> > > I'm write currently a HBase Java programm which iterates over every row
> > in
> > > a table. I have to modiy some rows if the column size (the amount of
> > > columns in this row) is bigger than 25000.
> > >
> > > Here is my sourcode: http://pastebin.com/njqG6ry6
> > >
> > > Is there any way to add a Filter to the scan Operation and load only
> rows
> > > where the size is bigger than 25k?
> > >
> > > Currently I check the size at the client, but therefore I have to load
> > > every row to the client site. It would be better if the wrong rows
> > already
> > > filtered at the "server" site.
> > >
> > > thanks
> > >
> > > John
> > >
> >
>

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: Linear Scalability in HBase

[Vladimir Rodionov]

You can not saturate region server with one client (unless you probably use 
hbase-async) if all data is cached in RAM.
In our performance tests we have run 10 clients (on different hosts) with 30 
threads each to max out 1 RS when all data
is in cache (block, page, etc).

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Ramu M S [ramu.ma...@gmail.com]
Sent: Friday, October 25, 2013 9:35 AM
To: user@hbase.apache.org
Subject: Re: Linear Scalability in HBase

Hi,

For me scalability is to achieve same throughput and latency with the
increase in number of clients.

In my case the data set increases with the number of clients. That's the
reason I vary both clients and region servers.

I'm trying to identify how the cluster should grow to handle data from more
clients so that the operations throughput and latency is under defined
limits.

Currently the limit is 15K OPS throughput and 1 ms latency.

To test, I have kept the data increase at around 15 million per server.

Each YCSB client actually runs 32 threads. So it is actually 15 million
more data for 32 more clients.

All machines are physical servers.

1) Read and write latency is around 1 ms in first whereas in second case
its little higher at 1.1 to 1.2 ms.

2) Keeping same number of clients as the first case, the latency reduced to
0.7 ms but throughput came down further to just 9K OPS

For the tests, I'm running both clients and Region servers on same machine.
But I tried in 8 Server scenario to run clients on different machines but
results were almost same as that of running clients on same machine.

Ganglia shows that system load is around 30% in both scenarios.

What I wanted to understand is how to grow the cluster to meet the needs of
both throughput and latency?

Regards,
Ramu

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: HBase wont responding when one region server goes down

[Vladimir Rodionov]

Google "HBase MTTR improvements in 0.96". 10-15 minutes of downtime is normal 
on versions prior to 0.96 with
default HBase configuration settings.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Sandeep L [sandeepvre...@outlook.com]
Sent: Monday, October 28, 2013 8:36 PM
To: user@hbase.apache.org
Subject: RE: HBase wont responding when one region server goes down

@Jean-mar
We are using HBase-0.94.1 with Hadoop-1.0.2.We have 15 Region servers, 5 
Zookeepers, 1 master, 3 backup masters and & 18 Dtatnodes.Our Java version is 
"jdk1.7.0_04"
The issue is whenever a region server is down scan/get is not working.
Thanks,Sandeep.

> From: jean-m...@spaggiari.org
> Date: Mon, 28 Oct 2013 15:55:31 -0400
> Subject: Re: HBase wont responding when one region server goes down
> To: user@hbase.apache.org
>
> Hi Sandeep,
>
> Do you have more information regarding your cluster? Number of nodes,
> hadoop version, what is "not responding", etc.?
>
> Thanks,
>
> JM
>
>
> 2013/10/28 Asaf Mesika 
>
> > If your query (scan) needs a region on the failed region server, the client
> > will fail and silently retry about 10 times. The sleep time increase as
> > each retry fails and can reach 10min. On the server side, the master takes
> > 3min to realize the RS failed thus issue a region move, which might take a
> > few minutes, which together explain the 10min query failure period
> >
> > On Monday, October 28, 2013, Sandeep L wrote:
> >
> > > Hi,
> > > We are using HBase 0.94.1.
> > > We are facing a strange issue form long time not at got any proper
> > > solution.
> > > Here is the issue:Whenever one region server in cluster goes down, whole
> > > HBase wont respond for some time (around 5 to 10 minutes).We are unable
> > to
> > > get any clue about the issue.
> > >
> > > Can some help us to resolve this issue.
> > >
> > > Thanks,Sandeep.
> >

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: HBase Client Performance Bottleneck in a Single Virtual Machine

[Vladimir Rodionov]

Michael, have you tried jstack on your client application?

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: michael.grund...@high5games.com [michael.grund...@high5games.com]
Sent: Sunday, November 03, 2013 7:46 PM
To: user@hbase.apache.org
Subject: HBase Client Performance Bottleneck in a Single Virtual Machine

Hi all; I posted this as a question on StackOverflow as well but realized I 
should have gone straight ot the horses-mouth with my question. Sorry for the 
double post!

We are running a series of HBase tests to see if we can migrate one of our 
existing datasets from a RDBMS to HBase. We are running 15 nodes with 5 
zookeepers and HBase 0.94.12 for this test.

We have a single table with three column families and a key that is 
distributing very well across the cluster. All of our queries are running a 
direct look-up; no searching or scanning. Since the HTablePool is now frowned 
upon, we are using the Apache commons pool and a simple connection factory to 
create a pool of connections and use them in our threads. Each thread creates 
an HTableInstance as needed and closes it when done. There are no leaks we can 
identify.

If we run a single thread and just do lots of random calls sequentially, the 
performance is quite good. Everything works great until we start trying to 
scale the performance. As we add more threads and try and get more work done in 
a single VM, we start seeing performance degrade quickly. The client code is 
simply attempting to run either one of several gets or a single put at a given 
frequency. It then waits until the next time to run, we use this to simulate 
the workload from external clients. With a single thread, we will see call 
times in the 2-3 milliseconds which is acceptable.

As we add more threads, this call time starts increasing quickly. What gets 
strange is if we add more VMs, the times hold steady across them all so clearly 
it's a bottleneck in the running instance and not the cluster. We can get a 
huge amount of processing happening across the cluster very easily - it just 
has to use a lot of VMs on the client side to do it. We know the contention 
isn't in the connection pool as we see the problem even when we have more 
connections than threads. Unfortunately, the times are spiraling out of control 
very quickly. We need it to support at least 128 threads in practice, but most 
important I want to support 500 updates/sec and 250 gets/sec. In theory, this 
should be a piece of cake for the cluster as we can do FAR more work than that 
with a few VMs, but we don't even get close to this with a single VM.

So my question: how do people building high-performance apps with HBase get 
around this? What approach are others using for connection pooling in a 
multi-threaded environment? There seems to be a surprisingly little amount of 
info about this on the web considering the popularity. Is there some client 
setting we need to use that makes it perform better in a threaded environment? 
We are going to try to cache HTable instances next but that's a total guess. 
There are solutions to offloading work to other VMs but we really want to avoid 
this as clearly the cluster can handle the load and it will dramatically 
decrease the application performance in critical areas.

Any help is greatly appreciated! Thanks!
-Mike

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: HBase Client Performance Bottleneck in a Single Virtual Machine

[Vladimir Rodionov]

It seems that there is a serious contention point inside standard HBase client 
API, that is why your application does not scale with multiple threads,
but scales with multiple JVMs. I would start with analyzing thread stack traces 
of your client application - you will easily spot excessive locking.  
 
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: michael.grund...@high5games.com [michael.grund...@high5games.com]
Sent: Monday, November 04, 2013 11:00 AM
To: user@hbase.apache.org
Subject: RE: HBase Client Performance Bottleneck in a Single Virtual Machine

Not yet, this is just a load test client. It literally does nothing but create 
threads to talk to HBase and run 4 different calls. Nothing else is done in the 
app at all.

To eliminate even more of our code from the loop, we just tried removing our 
connection pool entirely and just using a single connection per thread - no 
improvement. Then we tried creating the HTableInterface (all calls are against 
the same table) at the time of connection creation. The means thread to 
connection to table interface were all at 1 to 1 and not being passed around. 
No performance improvement.

Long story short, running a single thread it's fast. Start multithreading, it 
starts slowing down. CPU usage, memory usage, etc. are all negligible. The 
performance isn't terrible - it's probably good enough for the vast majority of 
users, but it's not good enough for our app. With one thread, it might take 5 
milliseconds. With 10 threads all spinning more quickly (40 milliseconds 
delay), the call time increases to 15-30 milliseconds. The problem is that at 
our throughput rates, that's a serious concern.

We are going to fire up a profiler next to see what we can find.

-Mike


-Original Message-
From: Vladimir Rodionov [mailto:vrodio...@carrieriq.com]
Sent: Monday, November 04, 2013 12:50 PM
To: user@hbase.apache.org
Subject: RE: HBase Client Performance Bottleneck in a Single Virtual Machine

Michael, have you tried jstack on your client application?

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: michael.grund...@high5games.com [michael.grund...@high5games.com]
Sent: Sunday, November 03, 2013 7:46 PM
To: user@hbase.apache.org
Subject: HBase Client Performance Bottleneck in a Single Virtual Machine

Hi all; I posted this as a question on StackOverflow as well but realized I 
should have gone straight ot the horses-mouth with my question. Sorry for the 
double post!

We are running a series of HBase tests to see if we can migrate one of our 
existing datasets from a RDBMS to HBase. We are running 15 nodes with 5 
zookeepers and HBase 0.94.12 for this test.

We have a single table with three column families and a key that is 
distributing very well across the cluster. All of our queries are running a 
direct look-up; no searching or scanning. Since the HTablePool is now frowned 
upon, we are using the Apache commons pool and a simple connection factory to 
create a pool of connections and use them in our threads. Each thread creates 
an HTableInstance as needed and closes it when done. There are no leaks we can 
identify.

If we run a single thread and just do lots of random calls sequentially, the 
performance is quite good. Everything works great until we start trying to 
scale the performance. As we add more threads and try and get more work done in 
a single VM, we start seeing performance degrade quickly. The client code is 
simply attempting to run either one of several gets or a single put at a given 
frequency. It then waits until the next time to run, we use this to simulate 
the workload from external clients. With a single thread, we will see call 
times in the 2-3 milliseconds which is acceptable.

As we add more threads, this call time starts increasing quickly. What gets 
strange is if we add more VMs, the times hold steady across them all so clearly 
it's a bottleneck in the running instance and not the cluster. We can get a 
huge amount of processing happening across the cluster very easily - it just 
has to use a lot of VMs on the client side to do it. We know the contention 
isn't in the connection pool as we see the problem even when we have more 
connections than threads. Unfortunately, the times are spiraling out of control 
very quickly. We need it to support at least 128 threads in practice, but most 
important I want to support 500 updates/sec and 250 gets/sec. In theory, this 
should be a piece of cake for the cluster as we can do FAR more work than that 
with a few VMs, but we don't even get close to this with a single VM.

So my question: how do people building high-performance apps with HBase get 
around this? What approach are others using for

RE: setCaching and setBatch

[Vladimir Rodionov]

I think, setBatch is  your OOME protection as since it limits the total amount 
of KeyValues
processed in a single call to StoreScanner.next.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Ted Yu [yuzhih...@gmail.com]
Sent: Monday, November 04, 2013 1:26 PM
To: user@hbase.apache.org
Subject: Re: setCaching and setBatch

bq. is there an OOME protections which lowers the caching in case ther
server is stressed in memory?

I am not aware of such mechanism.


On Mon, Nov 4, 2013 at 9:55 AM, Asaf Mesika  wrote:

> On this note:
> If I hammer a region server with scans that has let's say 100 rows caching
> - is there an OOME protections which lowers the caching in case ther server
> is stressed in memory? Or will it fail the region server with OOME?
>
> On Monday, November 4, 2013, Anoop John wrote:
>
> > Have u tested this?  AFAIK it happens the way u like.
> >
> > -Anoop-
> >
> > On Mon, Nov 4, 2013 at 10:49 PM, John  >
> > wrote:
> >
> > > Hi,
> > >
> > > If I have a question about the setCaching function. As I know the
> caching
> > > value influence how many rows will be cached in the memory of the
> region
> > > server. So for example I have 10 rows with 20k columns and I set a
> > scanner
> > > with the caching option 5 and the batch option 1k. What will hapen If I
> > > call the next() function?
> > >
> > > As far as I know, the region server will cache 5 rows with 1k columns
> > each
> > > in the memory. If thats right, is there a way to cache the batches
> > instead
> > > of the rows? In this example I would like to cache 5 batches from row
> one
> > > instead of 5 rows with each 1k columns.
> > >
> > > kind regards
> > >
> >
>

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: Scanner Caching with wildly varying row widths

[Vladimir Rodionov]

setBatch and setCaching are totally independent from each other. The latter one 
controls numbers of rows transferred from server to client in
one RPC call, the former one controls how many cells (key values) read from 
underlying storage per one call in HBase InternalScanner implementation.

To avoid OOME in Scan operation one should use setBatch with appropriate limit.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Dhaval Shah [prince_mithi...@yahoo.co.in]
Sent: Monday, November 04, 2013 3:10 PM
To: user@hbase.apache.org
Subject: Re: Scanner Caching with wildly varying row widths

You can use scan.setBatch() to limit the number of columns returned.. Note that 
it will split up a row into multiple rows from a client's perspective and 
client code might need to be modified to make use of the setBatch feature

Regards,
Dhaval



 From: Patrick Schless 
To: user 
Sent: Monday, 4 November 2013 6:03 PM
Subject: Scanner Caching with wildly varying row widths


We have an application where a row can contain anywhere between 1 and
360 cells (there's only 1 column family). In practice, most rows have
under 100 cells.

Now we want to run some mapreduce jobs that touch every cell within a range
(eg count how many cells we have).  With scanner caching set to something
like 250, the job will chug along for a long time, until it hits a row with
a lot of data, then it will die.  Setting the cache size down to 1 (row)
would presumably work, but take forever to run.

We have addressed this by writing some jobs that use coprocessors, which
allow us to pull back sets of cells instead of sets of rows, but this means
we can't use any of the built-in jobs that come with hbase (eg copyTable).
Is there any way around this? Have other people had to deal with such high
variability in their row sizes?

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: HBase Client Performance Bottleneck in a Single Virtual Machine

[Vladimir Rodionov]

One more: "hbase.ipc.client.tcpnodelay" set to true. It is worth trying as well.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: lars hofhansl [la...@apache.org]
Sent: Monday, November 04, 2013 5:55 PM
To: user@hbase.apache.org; lars hofhansl
Subject: Re: HBase Client Performance Bottleneck in a Single Virtual Machine

Here're one more thing to try. By default each HConnection will use a single 
TCP connection to multiplex traffic to each region server.

Try setting hbase.client.ipc.pool.size on the client to something > 1.

-- Lars




 From: lars hofhansl 
To: "user@hbase.apache.org" 
Sent: Monday, November 4, 2013 5:16 PM
Subject: Re: HBase Client Performance Bottleneck in a Single Virtual Machine


No. This is terrible.
If you can, please send a jstack and do some profiling. Is there an easy way to 
reproduce this with just a single RegionServer?
If so, I'd offer to do some profiling.

Thanks.


-- Lars





From: "michael.grund...@high5games.com" 
To: user@hbase.apache.org
Sent: Monday, November 4, 2013 11:00 AM
Subject: RE: HBase Client Performance Bottleneck in a Single Virtual Machine


Not yet, this is just a load test client. It literally does nothing but create 
threads to talk to HBase and run 4 different calls. Nothing else is done in the 
app at all.

To eliminate even more of our code from the loop, we just tried removing our 
connection pool entirely and just using a single connection per thread - no 
improvement. Then we tried creating the HTableInterface (all calls are against 
the same table) at the time of connection creation. The means thread to 
connection to table interface were all at 1 to 1 and not being passed around. 
No performance improvement.

Long story short, running a single thread it's fast. Start multithreading, it 
starts slowing down. CPU usage, memory usage, etc. are all negligible. The 
performance isn't terrible - it's probably good enough for the vast majority of 
users, but it's not good enough for our app. With one thread, it might take 5 
milliseconds. With 10 threads all spinning more quickly (40 milliseconds 
delay), the call time increases to 15-30 milliseconds. The problem is that at 
our throughput rates, that's a serious concern.

We are going to fire up a profiler next to see what we can find.

-Mike



-Original Message-
From: Vladimir Rodionov [mailto:vrodio...@carrieriq.com]
Sent: Monday, November 04, 2013 12:50 PM
To: user@hbase.apache.org
Subject: RE: HBase Client Performance Bottleneck in a Single Virtual Machine

Michael, have you tried jstack on your client application?

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: michael.grund...@high5games.com [michael.grund...@high5games.com]
Sent: Sunday, November 03, 2013 7:46 PM
To: user@hbase.apache.org
Subject: HBase Client Performance Bottleneck in a Single Virtual Machine

Hi all; I posted this as a question on StackOverflow as well but realized I 
should have gone straight ot the horses-mouth with my question. Sorry for the 
double post!

We are running a series of HBase tests to see if we can migrate one of our 
existing datasets from a RDBMS to HBase. We are running 15 nodes with 5 
zookeepers and HBase 0.94.12 for this test.

We have a single table with three column families and a key that is 
distributing very well across the cluster. All of our queries are running a 
direct look-up; no searching or scanning. Since the HTablePool is now frowned 
upon, we are using the Apache commons pool and a simple connection factory to 
create a pool of connections and use them in our threads. Each thread creates 
an HTableInstance as needed and closes it when done. There are no leaks we can 
identify.

If we run a single thread and just do lots of random calls sequentially, the 
performance is quite good. Everything works great until we start trying to 
scale the performance. As we add more threads and try and get more work done in 
a single VM, we start seeing performance degrade quickly. The client code is 
simply attempting to run either one of several gets or a single put at a given 
frequency. It then waits until the next time to run, we use this to simulate 
the workload from external clients. With a single thread, we will see call 
times in the 2-3 milliseconds which is acceptable.

As we add more threads, this call time starts increasing quickly. What gets 
strange is if we add more VMs, the times hold steady across them all so clearly 
it's a bottleneck in the running instance and not the cluster. We can get a 
huge amount of processing happening across the cluster very easily - it just 
has to use a lot of VMs on 

RE: How to change split policy of a specific existing table(not global)?‏

[Vladimir Rodionov]

Shel does not support all table/column families attributes in 0.94.6

This should work:

HBaseAdmin admin = new HBaseAdmin(conf);
HTableDescriptor desc = 
admin.getTableDescriptor(Bytes.toBytes(SPLIT_TEST_BIG));// get your table 
descriptor
admin.disableTable(SPLIT_TEST_BIG);
desc.setValue(HTableDescriptor.SPLIT_POLICY, 
"org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy");
admin.modifyTable(Bytes.toBytes(SPLIT_TEST_BIG), desc);
admin.enableTable(SPLIT_TEST_BIG);

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: WangRamon [ramon_w...@hotmail.com]
Sent: Tuesday, November 05, 2013 1:56 AM
To: user@hbase.apache.org
Subject: How to change split policy of a specific existing table(not global)?‏

Hi Folks
I'm trying to change the default split policy of an existing table in HBase 
0.94.6-cdh4.4.0. The way i think should work is by using HBase Shell, so i 
executed the following command in the SHELL:
disable 'SPLIT_TEST_BIG'alter 'SPLIT_TEST_BIG', {METHOD => 'table_att', 
'SPLIT_POLICY' => 
'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy'}alter 
'SPLIT_TEST_BIG', {METHOD => 'table_att', 'MAX_FILESIZE' => 
'107374182400'}enable 'SPLIT_TEST_BIG'

But it doesn't work, only the 'MAX_FILESIZE' property get changed, the table 
still uses the default split policy which is 
'IncreasingToUpperBoundRegionSplitPolicy', so can anyone tell me what is the 
correct way to do it, thanks!
CheersRamon

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: How to change split policy of a specific existing table(not global)?‏

[Vladimir Rodionov]

There is a ticket for that :)

https://issues.apache.org/jira/browse/HBASE-9218

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: saint@gmail.com [saint@gmail.com] On Behalf Of Stack 
[st...@duboce.net]
Sent: Tuesday, November 05, 2013 7:53 AM
To: Hbase-User
Subject: Re: How to change split policy of a specific existing table(not 
global)?‏

On Tue, Nov 5, 2013 at 6:04 AM, WangRamon  wrote:

> Hi JM
> Thanks for your help, yes, as you said the syntax is totally correct as it
> can be executed in HBase Shell without any errors.
> By checking the source code of class HRegion which initialise the region
> policy from HTableDescriptor, not sure if that SPLIT_POLICY value can be
> dynamically changed after a table(and its first child region) is created?
>
> > 1) If you look in the Web UI, do you see you new attribute set for this
> > table?
> I don't know there is a specific UI which i can view the new attributes
> except the master-status UI, if that's the UI you are talking about, i can
> only see the MAX_FILESIZE property is set correctly:{NAME =>
> 'SPLIT_TEST_BIG', MAX_FILESIZE => '107374182400', FAMILIES => [{NAME =>
> 'f1'}]}
>

What JMS said.

Seems like you are looking at the right place in the UI.

After altering the table, if you do

hbase> describe "SPLIT_TEST_BIG"

... do you see the new attribute set?

If not, the shell must be not setting this attribute properly (bug).

St.Ack





> > 2) How have you verified that it doesn't work? What do you have on the
> logs?
> I use YCSB to insert data into that table, and i did saw there are some
> new regions created after i running the test for a while. And here is the
> LOG i get from one of the region severs:
> 2013-11-05 08:34:46,571 DEBUG
> org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy:
> ShouldSplit because f1 size=414926238, sizeToCheck=134217728,
> regionsWithCommonTable=1
> Any idea?
> ThanksRamon
>
> > From: jean-m...@spaggiari.org
> > Date: Tue, 5 Nov 2013 07:40:04 -0500
> > Subject: Re: How to change split policy of a specific existing table(not
> global)?‏
> > To: user@hbase.apache.org
> >
> > This seems to be correct.
> >
> > 2 questions.
> >
> > 1) If you look in the Web UI, do you see you new attribute set for this
> > table?
> > 2) How have you verified that it doesn't work? What do you have on the
> logs?
> >
> > JM
> >
> >
> > 2013/11/5 WangRamon 
> >
> > > Hi Folks
> > > I'm trying to change the default split policy of an existing table in
> > > HBase 0.94.6-cdh4.4.0. The way i think should work is by using HBase
> Shell,
> > > so i executed the following command in the SHELL:
> > > disable 'SPLIT_TEST_BIG'alter 'SPLIT_TEST_BIG', {METHOD => 'table_att',
> > > 'SPLIT_POLICY' =>
> > >
> 'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy'}alter
> > > 'SPLIT_TEST_BIG', {METHOD => 'table_att', 'MAX_FILESIZE' =>
> > > '107374182400'}enable 'SPLIT_TEST_BIG'
> > >
> > > But it doesn't work, only the 'MAX_FILESIZE' property get changed, the
> > > table still uses the default split policy which is
> > > 'IncreasingToUpperBoundRegionSplitPolicy', so can anyone tell me what
> is
> > > the correct way to do it, thanks!
> > > CheersRamon
>
>

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: HBase Delete - based on non-rowkey columns

[Vladimir Rodionov]

BulkDeleteEndpoint coprocessor to your help. Its the endpoint coprocessor. 
Refer HBase 0.94+ API and HBase book on
how to use it for bulk delete operations.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: raviprasa...@polarisft.com [raviprasa...@polarisft.com]
Sent: Wednesday, November 06, 2013 1:14 AM
To: user@hbase.apache.org
Subject: HBase Delete - based on non-rowkey columns

Hi all,
I have two questions
1)   In HBASE, is it possible to delete a row  based on a column name &  value. 
  The column name should not be a row key, because in this scenario we cannot 
deletebased on the row key.


2) Is it possible to delete a row in HBase table , based on the partial row key.

Thanks in advance.



Regards
Raviprasad. T


This e-Mail may contain proprietary and confidential information and is sent 
for the intended recipient(s) only.  If by an addressing or transmission error 
this mail has been misdirected to you, you are requested to delete this mail 
immediately. You are also hereby notified that any use, any form of 
reproduction, dissemination, copying, disclosure, modification, distribution 
and/or publication of this e-mail message, contents or its attachment other 
than by its intended recipient/s is strictly prohibited.

Visit us at http://www.polarisFT.com

Confidentiality Notice:  The information contained in this message, including 
any attachments hereto, may be confidential and is intended to be read only by 
the individual or entity to whom this message is addressed. If the reader of 
this message is not the intended recipient or an agent or designee of the 
intended recipient, please note that any review, use, disclosure or 
distribution of this message or its attachments, in any form, is strictly 
prohibited.  If you have received this message in error, please immediately 
notify the sender and/or notificati...@carrieriq.com and delete or destroy any 
copy of this message and its attachments.

RE: HBase Client Performance Bottleneck in a Single Virtual Machine

[Vladimir Rodionov]

Yes, this issue exists definitely. I do remember my own thread on dev-list a 
while back when I tried to saturate on RS with all data cached. It took me 10 
client JVM and a couple hundreds threads.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: lars hofhansl [la...@apache.org]
Sent: Wednesday, November 06, 2013 11:09 AM
To: user@hbase.apache.org
Subject: Re: HBase Client Performance Bottleneck in a Single Virtual Machine

If you get the chance, please send a jstack of the client when it is slow, or 
some profile data if you have, or just some code reproducing the problem.
We should track this down.

-- Lars




 From: "michael.grund...@high5games.com" 
To: user@hbase.apache.org
Sent: Tuesday, November 5, 2013 8:35 AM
Subject: RE: HBase Client Performance Bottleneck in a Single Virtual Machine


I'm finding this configuration process very confusing. This is what we are 
doing in Java code:

Configuration configuration = HBaseConfiguration.create();
configuration.set("hbase.zookeeper.quorum", 
props.getProperty(HBASE_NODES));
configuration.set("zookeeper.znode.parent", 
props.getProperty(HBASE_PARAMS));
configuration.setBoolean(" hbase.ipc.client.tcpnodelay", true);
configuration.setInt("hbase.client.ipc.pool.size", 10);

Adding those last two lines has made no improvement. Likewise, updating 
hbase-site.xml on the cluster to include these settings has made no 
improvement. The problem is that I have zero confidence it's actually taking 
the settings in the first place.

-Mike



-Original Message-
From: lars hofhansl [mailto:la...@apache.org]
Sent: Tuesday, November 05, 2013 9:47 AM
To: user@hbase.apache.org
Subject: Re: HBase Client Performance Bottleneck in a Single Virtual Machine

Hi Mike,

hbase-site.xml has both client and server side settings.

In your client, where do you set the ZK quorum of the cluster? At that same 
spot you'd set these options.

Assuming you're running the clients with classes from a normal HBase install, 
you would add these setting to the hbase-site.xml there.

-- Lars




From: "michael.grund...@high5games.com" 
To: user@hbase.apache.org
Sent: Tuesday, November 5, 2013 6:43 AM
Subject: RE: HBase Client Performance Bottleneck in a Single Virtual Machine


Thanks for the input, we'll do some more testing on this today. Where do these 
settings get made? In the configuration used to create the pool or somewhere 
else? It appears hbase-site.xml "on the client" but we have nothing like that 
so I think I'm misunderstanding something. Thanks!

-Mike



-Original Message-
From: Vladimir Rodionov [mailto:vrodio...@carrieriq.com]
Sent: Monday, November 04, 2013 8:41 PM
To: user@hbase.apache.org; lars hofhansl
Subject: RE: HBase Client Performance Bottleneck in a Single Virtual Machine

One more: "hbase.ipc.client.tcpnodelay" set to true. It is worth trying as well.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: lars hofhansl [la...@apache.org]
Sent: Monday, November 04, 2013 5:55 PM
To: user@hbase.apache.org; lars hofhansl
Subject: Re: HBase Client Performance Bottleneck in a Single Virtual Machine

Here're one more thing to try. By default each HConnection will use a single 
TCP connection to multiplex traffic to each region server.

Try setting hbase.client.ipc.pool.size on the client to something > 1.

-- Lars




From: lars hofhansl 
To: "user@hbase.apache.org" 
Sent: Monday, November 4, 2013 5:16 PM
Subject: Re: HBase Client Performance Bottleneck in a Single Virtual Machine


No. This is terrible.
If you can, please send a jstack and do some profiling. Is there an easy way to 
reproduce this with just a single RegionServer?
If so, I'd offer to do some profiling.

Thanks.


-- Lars





From: "michael.grund...@high5games.com" 
To: user@hbase.apache.org
Sent: Monday, November 4, 2013 11:00 AM
Subject: RE: HBase Client Performance Bottleneck in a Single Virtual Machine


Not yet, this is just a load test client. It literally does nothing but create 
threads to talk to HBase and run 4 different calls. Nothing else is done in the 
app at all.

To eliminate even more of our code from the loop, we just tried removing our 
connection pool entirely and just using a single connection per thread - no 
improvement. Then we tried creating the HTableInterface (all calls are against 
the same table) at the time of connection creation. The means thread to 
connection to table interface were all at 1 to 1 and not being passe

RE: HBase Client Performance Bottleneck in a Single Virtual Machine

[Vladimir Rodionov]

This thread describes my similar findings:

http://mail-archives.apache.org/mod_mbox/hbase-dev/201307.mbox/%3CDC5EBE7F3610EB4CA5C7E92D76873E1517ECF1DCCE%40exchange2007.carrieriq.com%3E

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: Vladimir Rodionov
Sent: Wednesday, November 06, 2013 11:43 AM
To: user@hbase.apache.org; lars hofhansl
Subject: RE: HBase Client Performance Bottleneck in a Single Virtual Machine

Yes, this issue exists definitely. I do remember my own thread on dev-list a 
while back when I tried to saturate on RS with all data cached. It took me 10 
client JVM and a couple hundreds threads.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: lars hofhansl [la...@apache.org]
Sent: Wednesday, November 06, 2013 11:09 AM
To: user@hbase.apache.org
Subject: Re: HBase Client Performance Bottleneck in a Single Virtual Machine

If you get the chance, please send a jstack of the client when it is slow, or 
some profile data if you have, or just some code reproducing the problem.
We should track this down.

-- Lars




 From: "michael.grund...@high5games.com" 
To: user@hbase.apache.org
Sent: Tuesday, November 5, 2013 8:35 AM
Subject: RE: HBase Client Performance Bottleneck in a Single Virtual Machine


I'm finding this configuration process very confusing. This is what we are 
doing in Java code:

Configuration configuration = HBaseConfiguration.create();
configuration.set("hbase.zookeeper.quorum", 
props.getProperty(HBASE_NODES));
configuration.set("zookeeper.znode.parent", 
props.getProperty(HBASE_PARAMS));
configuration.setBoolean(" hbase.ipc.client.tcpnodelay", true);
configuration.setInt("hbase.client.ipc.pool.size", 10);

Adding those last two lines has made no improvement. Likewise, updating 
hbase-site.xml on the cluster to include these settings has made no 
improvement. The problem is that I have zero confidence it's actually taking 
the settings in the first place.

-Mike



-Original Message-
From: lars hofhansl [mailto:la...@apache.org]
Sent: Tuesday, November 05, 2013 9:47 AM
To: user@hbase.apache.org
Subject: Re: HBase Client Performance Bottleneck in a Single Virtual Machine

Hi Mike,

hbase-site.xml has both client and server side settings.

In your client, where do you set the ZK quorum of the cluster? At that same 
spot you'd set these options.

Assuming you're running the clients with classes from a normal HBase install, 
you would add these setting to the hbase-site.xml there.

-- Lars




From: "michael.grund...@high5games.com" 
To: user@hbase.apache.org
Sent: Tuesday, November 5, 2013 6:43 AM
Subject: RE: HBase Client Performance Bottleneck in a Single Virtual Machine


Thanks for the input, we'll do some more testing on this today. Where do these 
settings get made? In the configuration used to create the pool or somewhere 
else? It appears hbase-site.xml "on the client" but we have nothing like that 
so I think I'm misunderstanding something. Thanks!

-Mike



-Original Message-
From: Vladimir Rodionov [mailto:vrodio...@carrieriq.com]
Sent: Monday, November 04, 2013 8:41 PM
To: user@hbase.apache.org; lars hofhansl
Subject: RE: HBase Client Performance Bottleneck in a Single Virtual Machine

One more: "hbase.ipc.client.tcpnodelay" set to true. It is worth trying as well.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com


From: lars hofhansl [la...@apache.org]
Sent: Monday, November 04, 2013 5:55 PM
To: user@hbase.apache.org; lars hofhansl
Subject: Re: HBase Client Performance Bottleneck in a Single Virtual Machine

Here're one more thing to try. By default each HConnection will use a single 
TCP connection to multiplex traffic to each region server.

Try setting hbase.client.ipc.pool.size on the client to something > 1.

-- Lars




From: lars hofhansl 
To: "user@hbase.apache.org" 
Sent: Monday, November 4, 2013 5:16 PM
Subject: Re: HBase Client Performance Bottleneck in a Single Virtual Machine


No. This is terrible.
If you can, please send a jstack and do some profiling. Is there an easy way to 
reproduce this with just a single RegionServer?
If so, I'd offer to do some profiling.

Thanks.


-- Lars





From: "michael.grund...@high5games.com" 
To: user@hbase.apache.org
Sent: Monday, November 4, 2013 11:00 AM
Subject: RE: HBase Client Performance Bottleneck in a Single Virtual Machine


Not yet, this is

Re: Hbase ImportTSV runs in parallel?

[Vladimir Rodionov]

Search google "how to run jobs in parallel in Hadoop"

Your mapreduce configuration allows you to run  one job at a time. This
usually happens
when number of job's tasks exceeds capacity of a cluster.

-Vlad

On Thu, Mar 5, 2015 at 3:03 PM, Siva  wrote:

> Hi All,
>
>
>
> I’m loading data to Hbase by using Hbase ImportTsv utility. When I kick off
> this process simultaneously for different tables in different sessions,
> both the process starts in parallel till it reaches the map reduce program.
> Once one of the process kicks off map reduce job for one table, other
> process waits till the first one finishes.
>
>
>
> Both the processes executes in parallel till it reaches MR phase, from
> there onwards they becomes sequential. Is there any limitation in Hbase
> that only one table should load through bulk loader at a time or is there a
> property in hbase that controls this feature?
>
>
>
> I want to load the different tables at same time by using bulk loader. Any
> help in this regard is appreciated.
>
>
>
> Thanks in advance.
>
>
>
> Thanks,
>
> Siva.
>

Re: avoiding hot spot for timestamp prefix key

[Vladimir Rodionov]

RegionSplitPolicy only allows you to customize split point (row key). All
rows above this split point will go to
the first daughter region, below - to the second.

The answer on original question is - No, you can not have your custom
policy based on a second part of a key.

-Vlad

On Fri, May 22, 2015 at 2:43 AM, Michael Segel 
wrote:

> This is why I created HBASE-12853.
>
> So you don’t have to specify a custom split policy.
>
> Of course the simple solutions are often passed over because of NIH.  ;-)
>
> To be blunt… You encapsulate the bucketing code so that you have a single
> API in to HBase regardless of the type of storage underneath.
> KISS is maintained and you stop people from attempting to do stupid
> things.   (cc’ing dev@hbase) As a product owner, (read PMC / committers)
> you want to keep people from mucking about in the internals.  While its
> true that its open source, and you will have some who want to muck around,
> you also have to consider the corporate users who need something that is
> reliable and less customized so that its supportable.  This is the vendor’s
> dilemma. (hint Cloudera , Horton, IBM, MapR)  You’re selling support to
> HBase and if a customer starts to overload internals with their own code,
> good luck in supporting it.  This is why you do things like 12853 because
> it makes your life easier.
>
> This isn’t a sexy solution. Its core engineering work.
>
> HTH
>
> -Mike
>
> > On May 22, 2015, at 4:22 AM, Shushant Arora 
> wrote:
> >
> > since custom split policy is based on second part i.e guid so key with
> > first part as 2015-05-22 00:01:02 will be in which region how will that
> be
> > identified?
> >
> >
> > On Fri, May 22, 2015 at 1:12 PM, Ted Yu  wrote:
> >
> >> The custom split policy needs to respect the fact that timestamp is the
> >> leading part of the rowkey.
> >>
> >> This would avoid the overlap you mentioned.
> >>
> >> Cheers
> >>
> >>
> >>
> >>> On May 21, 2015, at 11:55 PM, Shushant Arora <
> shushantaror...@gmail.com>
> >> wrote:
> >>>
> >>> guid change with every key, patterns is
> >>> 2015-05-22 00:02:01#AB12EC945
> >>> 2015-05-22 00:02:02#CD9870001234AB457
> >>>
> >>> When we specify custom split algorithm , it may happen that keys of
> same
> >>> sorting order range say (1-7) lies in region R1 as well as in region
> R2?
> >>> Then how .META. table will make further lookups at read time,  say I
> >> search
> >>> for key 3, then will it search in both the regions R1 and R2 ?
> >>>
>  On Fri, May 22, 2015 at 10:48 AM, Ted Yu  wrote:
> 
>  Does guid change with every key ?
> 
>  bq. use second part of key
> 
>  I don't think so. Suppose first row in the parent region is
>  '1432104178817#321'. After split, the first row in first daughter
> region
>  would still be '1432104178817#321'. Right ?
> 
>  Cheers
> 
>  On Thu, May 21, 2015 at 9:57 PM, Shushant Arora <
> >> shushantaror...@gmail.com
>  wrote:
> 
> > Can I avoid hotspot of region with custom region split policy in
> hbase
> >> 0.96 .
> >
> > Key is of the form timestamp#guid.
> > So can I have custom region split policy and use second part of key
> >> (i.e)
> > guid as region split criteria and avoid hot spot??
> 
> >>
>
>

Re: HBase snapshots and Compaction

[Vladimir Rodionov]

Its possible that logic of ExploringCompactionPolicy (default compaction
policy) is broken. I am looking into this code (master):

private boolean isBetterSelection(List bestSelection,
  long bestSize, List selection, long size, boolean
mightBeStuck) {
if (mightBeStuck && bestSize > 0 && size > 0) {
  // Keep the selection that removes most files for least size. That
penaltizes adding
  // large files to compaction, but not small files, so we don't become
totally inefficient
  // (might want to tweak that in future). Also, given the current
order of looking at
  // permutations, prefer earlier files and smaller selection if the
difference is small.
  final double REPLACE_IF_BETTER_BY = 1.05;
  double thresholdQuality = ((double)bestSelection.size() / bestSize) *
REPLACE_IF_BETTER_BY;
  return thresholdQuality < ((double)selection.size() / size);
}
// Keep if this gets rid of more files.  Or the same number of files
for less io.
return selection.size() > bestSelection.size()
  || (selection.size() == bestSelection.size() && size < bestSize);
  }

which compares two selections and what I see here is when mightBeStuck =
false selection with more files will
always be preferred.

Correct if I am wrong.

-Vlad

On Thu, May 28, 2015 at 8:00 AM, Akmal Abbasov 
wrote:

> Hi Ted,
> Thank you for reply.
> Yes, it was promoted to a major compaction because all files were
> eligible, but the thing I don’t understand, is why all of them were
> eligible?
> afaik, the compaction algorithm should select the best match for
> compaction, and it should include files with similar sizes.
> But as you can see from the logs the files selected have: 4.7K, 5.1K, 3.8K
> and 10.8M.
> Why it is including 10.8M file?
> Which setting should be tuned to avoid this?
> Thank you.
>
> Kind regards,
> Akmal Abbasov
>
> > On 28 May 2015, at 16:54, Ted Yu  wrote:
> >
> > bq. Completed major compaction of 4 file(s) in s of metrics,
> > V\xA36\x56\x5E\xC5}\xA1\x43\x00\x32\x00T\x1BU\xE0,
> > 3547f43afae5ac3f4e8a162d43a892b4.1417707276446.
> >
> > The compaction involved all the files of store 's' for the region. Thus
> it
> > was considered major compaction.
> >
> > Cheers
> >
> > On Thu, May 28, 2015 at 2:16 AM, Akmal Abbasov  >
> > wrote:
> >
> >> Hi Ted,
> >> Sorry for a late reply.
> >> Here is a snippet from log file
> >> 2015-05-28 00:54:39,754 DEBUG
> >> [regionserver60020-smallCompactions-1432714643311]
> >> regionserver.CompactSplitThread: CompactSplitThread Status:
> >> compaction_queue=(0:27), split_queue=0, merge_queue=0
> >> 2015-05-28 00:54:39,754 DEBUG
> >> [regionserver60020-smallCompactions-1432714643311]
> >> compactions.RatioBasedCompactionPolicy: Selecting compaction from 4
> store
> >> files, 0 compacting, 4 eligible, 10 blocking
> >> 2015-05-28 00:54:39,755 DEBUG
> >> [regionserver60020-smallCompactions-1432714643311]
> >> compactions.ExploringCompactionPolicy: Exploring compaction algorithm
> has
> >> selected 4 files of size 11304175 starting at candidate #0 after
> >> considering 3 permutations with 3 in ratio
> >> 2015-05-28 00:54:39,755 DEBUG
> >> [regionserver60020-smallCompactions-1432714643311] regionserver.HStore:
> >> 3547f43afae5ac3f4e8a162d43a892b4 - s: Initiating major compaction
> >> 2015-05-28 00:54:39,755 INFO
> >> [regionserver60020-smallCompactions-1432714643311] regionserver.HRegion:
> >> Starting compaction on s in region
> >>
> metrics,V\xA36\x56\x5E\xC5}\xA1\x43\x00\x32\x00T\x1BU\xE0,1417707276446.3547f43afae5ac3f4e8a162d43a892b4.
> >> 2015-05-28 00:54:39,755 INFO
> >> [regionserver60020-smallCompactions-1432714643311] regionserver.HStore:
> >> Starting compaction of 4 file(s) in s of
> >>
> metrics,V\xA36\x56\x5E\xC5}\xA1\x43\x00\x32\x00T\x1BU\xE0,1417707276446.3547f43afae5ac3f4e8a162d43a892b4.
> >> into
> >>
> tmpdir=hdfs://prod1/hbase/data/default/metrics/3547f43afae5ac3f4e8a162d43a892b4/.tmp,
> >> totalSize=10.8 M
> >> 2015-05-28 00:54:39,756 DEBUG
> >> [regionserver60020-smallCompactions-1432714643311]
> compactions.Compactor:
> >> Compacting
> >>
> hdfs://prod1/hbase/data/default/metrics/3547f43afae5ac3f4e8a162d43a892b4/s/dab3e768593e44a39097451038c5ebd0,
> >> keycount=3203, bloomtype=ROW, size=10.8 M, encoding=NONE,
> seqNum=172299974,
> >> earliestPutTs=1407941317178
> >> 2015-05-28 00:54:39,756 DEBUG
> >> [regionserver60020-smallCompactions-1432714643311]
> compactions.Compactor:
> >> Compacting
> >>
> hdfs://prod1/hbase/data/default/metrics/3547f43afae5ac3f4e8a162d43a892b4/s/2d6472ef99a5478689f7ba822bc407a7,
> >> keycount=4, bloomtype=ROW, size=4.7 K, encoding=NONE, seqNum=172299976,
> >> earliestPutTs=1432761158066
> >> 2015-05-28 00:54:39,756 DEBUG
> >> [regionserver60020-smallCompactions-1432714643311]
> compactions.Compactor:
> >> Compacting
> >>
> hdfs://prod1/hbase/data/default/metrics/3547f43afae5ac3f4e8a162d43a892b4/s/bdbc806d045740e69ab34e3ea2e113c4,
> >> keycount=6, bloomtype=ROW, size=5.1 K, encoding=NONE, seqNum=172299977,
> >> ea

Re: Response Too Slow in RegionServer Logs

[Vladimir Rodionov]

Is there any reason for 27G heap? It seems you run M/R job? If yes, then I
would recommend you trying M/R over snapshots.  Combination of large HBase
heap and M/R is very hard to tune if possible at all
You can also try reducing number of map tasks and check your MR job
resource consumption
On May 29, 2015 8:58 AM, "Bryan Beaudreault" 
wrote:

> That one long GC aside, look at the timings of the others as well.  Even
> the smaller GCs are taking up the majority of each second.
>
> For a heap that size you might want to try a java version over java7u60 and
> use the G1GC.  Otherwise there are a bunch of resources on the web
> including in the refguide for how to tune GCs for hbase.
>
>
> On Fri, May 29, 2015 at 11:49 AM, rahul malviya <
> malviyarahul2...@gmail.com>
> wrote:
>
> > This is a sample from gc log file. At the end I see long gc pauses. Is
> > there a way I can tune this ?
> >
> > 2015-04-29T22:46:12.387+: 98061.660: [GC2015-04-29T22:46:12.387+:
> > 98061.661: [ParNew: 572757K->63867K(580608K), 0.6549550 secs]
> > 13294553K->12811090K(20001132K), 0.6551600 secs] [Times: user=8.09
> > sys=0.07, real=0.65 secs]
> > 2015-04-29T22:46:13.165+: 98062.438: [GC2015-04-29T22:46:13.165+:
> > 98062.439: [ParNew: 579901K->55337K(580608K), 1.1941730 secs]
> > 13327124K->12846913K(20001132K), 1.1944150 secs] [Times: user=5.39
> > sys=0.08, real=1.20 secs]
> > 2015-04-29T22:46:14.485+: 98063.759: [GC2015-04-29T22:46:14.485+:
> > 98063.759: [ParNew: 571496K->62270K(580608K), 0.8990460 secs]
> > 13363071K->12891412K(20001132K), 0.8992880 secs] [Times: user=7.07
> > sys=0.05, real=0.90 secs]
> > 2015-04-29T22:46:15.519+: 98064.793: [GC2015-04-29T22:46:15.519+:
> > 98064.793: [ParNew: 578417K->41496K(580608K), 0.8849310 secs]
> > 13407559K->12907053K(20001132K), 0.8851300 secs] [Times: user=4.98
> > sys=0.08, real=0.88 secs]
> > 2015-04-29T22:46:16.547+: 98065.821: [GC2015-04-29T22:46:16.547+:
> > 98065.821: [ParNew: 557652K->50791K(580608K), 0.6345050 secs]
> > 13423210K->12940694K(20001132K), 0.6347240 secs] [Times: user=7.99
> > sys=0.04, real=0.63 secs]
> > 2015-04-29T22:46:17.273+: 98066.547: [GC2015-04-29T22:46:17.273+:
> > 98066.547: [ParNew: 566889K->48425K(580608K), 0.6372610 secs]
> > 13456792K->12966994K(20001132K), 0.6375250 secs] [Times: user=3.72
> > sys=0.05, real=0.64 secs]
> > 2015-04-29T22:46:18.035+: 98067.309: [GC2015-04-29T22:46:18.035+:
> > 98067.309: [ParNew: 56K->52607K(580608K), 0.7116560 secs]
> > 13483013K->12997986K(20001132K), 0.7119250 secs] [Times: user=7.16
> > sys=0.05, real=0.72 secs]
> > 2015-04-29T22:46:18.875+: 98068.149: [GC2015-04-29T22:46:18.875+:
> > 98068.149: [ParNew: 568635K->54078K(580608K), 0.5286160 secs]
> > 13514014K->13026734K(20001132K), 0.5288800 secs] [Times: user=4.70
> > sys=0.05, real=0.53 secs]
> > 2015-04-29T22:46:19.521+: 98068.795: [GC2015-04-29T22:46:19.521+:
> > 98068.795: [ParNew: 570106K->64416K(580608K), 1.0709370 secs]
> > 13542763K->13091755K(20001132K), 1.0712320 secs] [Times: user=9.57
> > sys=0.08, real=1.08 secs]
> > 2015-04-29T22:46:20.751+: 98070.024: [GC2015-04-29T22:46:20.751+:
> > 98070.024: [ParNew: 580387K->47393K(580608K), 0.5688620 secs]
> > 13607725K->13099551K(20001132K), 0.5690960 secs] [Times: user=6.48
> > sys=0.05, real=0.57 secs]
> > 2015-04-29T22:46:21.393+: 98070.667: [GC2015-04-29T22:46:21.393+:
> > 98070.667: [ParNew: 563351K->26473K(580608K), 0.7063100 secs]
> > 13615509K->13429978K(20001132K), 0.7065560 secs] [Times: user=10.49
> > sys=0.07, real=0.71 secs]
> > 2015-04-29T22:46:22.214+: 98071.487: [GC2015-04-29T22:46:22.214+:
> > 98071.488: [ParNew: 292415K->39128K(580608K), 0.0293500
> > secs]2015-04-29T22:46:22.243+: 98071.517: [CMS:
> > 13403504K->7807614K(19420524K), 284.4275150 secs]
> > 13695919K->7807614K(20001132K), [CMS Perm : 30509K->30432K(51000K)],
> > 284.4594370 secs] [Times: user=5.55 sys=2.50, real=284.42 secs]
> > Heap
> >  par new generation   total 2146944K, used 868806K [0x00016360,
> > 0x0001f4f9, 0x0001f4f9)
> >   eden space 1908416K,  45% used [0x00016360, 0x0001986718f8,
> > 0x0001d7db)
> >   from space 238528K,   0% used [0x0001d7db, 0x0001d7db,
> > 0x0001e66a)
> >   to   space 238528K,   0% used [0x0001e66a, 0x0001e66a,
> > 0x0001f4f9)
> >  concurrent mark-sweep generation total 19420524K, used 7807614K
> > [0x0001f4f9, 0x0006964eb000, 0x0007fae0)
> >  concurrent-mark-sweep perm gen total 51000K, used 30498K
> > [0x0007fae0, 0x0007fdfce000, 0x0008)
> >
> > Thanks,
> > Rahul
> >
> >
> > On Fri, May 29, 2015 at 8:41 AM, Bryan Beaudreault <
> > bbeaudrea...@hubspot.com
> > > wrote:
> >
> > > >
> > > > 2014-08-14 21:35:16,740 WARN org.apache.hadoop.hbase.util.Sleeper: We
> > > > slept
> > > > 14912ms instead of 3000ms, this is likely due to a long garbage
> > > colle

Re: Hbase vs Cassandra

[Vladimir Rodionov]

Couple more + for HBase

* Coprocessor framework (custom code inside Region Server and Master
Servers), which Cassandra is missing, afaik.
   Coprocessors have been widely used by hBase users (Phoenix SQL, for
example) since inception (in 0.92).
* HBase security model is more mature and align well with Hadoop/HDFS
security. Cassandra provides just basic authentication/authorization/SSL
encryption, no Kerberos, no end-to-end data encryption, no cell level
security.

-Vlad

On Sun, May 31, 2015 at 12:05 PM, lars hofhansl  wrote:

> You really have to try out both if you want to be sure.
>
> The fundamental differences that come to mind are:
> * HBase is always consistent. Machine outages lead to inability to read or
> write data on that machine. With Cassandra you can always write.
>
> * Cassandra defaults to a random partitioner, so range scans are not
> possible (by default)
> * HBase has a range partitioner (if you don't want that the client has to
> prefix the rowkey with a prefix of a hash of the rowkey). The main feature
> that set HBase apart are range scans.
>
> * HBase is much more tightly integrated with Hadoop/MapReduce/HDFS, etc.
> You can map reduce directly into HFiles and map those into HBase instantly.
>
> * Cassandra has a dedicated company supporting (and promoting) it.
> * Getting started is easier with Cassandra. For HBase you need to run HDFS
> and Zookeeper, etc.
> * I've heard lots of anecdotes about Cassandra working nicely with small
> cluster (< 50 nodes) and quick degenerating above that.
> * HBase does not have a query language (but you can use Phoenix for full
> SQL support)
> * HBase does not have secondary indexes (having an eventually consistent
> index, similar to what Cassandra has, is easy in HBase, but making it as
> consistent as the rest of HBase is hard)
>
> * Everything you'll hear here is biased :)
>
>
>
> From personal experience... At Salesforce we spent a few months
> prototyping various stores (including Cassandra) and arrived at HBase. Your
> mileage may vary.
>
>
> -- Lars
>
>
> - Original Message -
> From: Ajay 
> To: user@hbase.apache.org
> Cc:
> Sent: Friday, May 29, 2015 12:12 PM
> Subject: Hbase vs Cassandra
>
> Hi,
>
> I need some info on Hbase vs Cassandra as a data store (in general plus
> specific to time series data).
>
> The comparison in the following helps:
> 1: features
> 2: deployment and monitoring
> 3: performance
> 4: anything else
>
> Thanks
> Ajay
>

Re: How to scan only Memstore from end point co-processor

[Vladimir Rodionov]

InternalScan has ctor from Scan object

See https://issues.apache.org/jira/browse/HBASE-12720

You can instantiate InternalScan from Scan, set checkOnlyMemStore, then
open RegionScanner, but the best approach is
to cache data on write and run regular RegionScanner from memstore and
block cache.

best,
-Vlad




On Sun, May 31, 2015 at 11:45 PM, Anoop John  wrote:

> If your scan is having a time range specified in it, HBase internally will
> check this against the time range of files etc and will avoid those which
> are clearly out of your interested time range.  You dont have to do any
> thing for this.  Make sure you set the TimeRange for ur read
>
> -Anoop-
>
> On Mon, Jun 1, 2015 at 12:09 PM, ramkrishna vasudevan <
> ramkrishna.s.vasude...@gmail.com> wrote:
>
> > We have a postScannerOpen hook in the CP but that may not give you a
> direct
> > access to know which one are the internal scanners on the Memstore and
> > which one are on the store files. But this is possible but we may need to
> > add some new hooks at this place where we explicitly add the internal
> > scanners required for a scan.
> >
> > But still a general question - are you sure that your data will be only
> in
> > the memstore and that the latest data would not have been flushed by that
> > time from your memstore to the Hfiles.  I see that your scenario is write
> > centric and how can you guarentee your data can be in memstore only?
> > Though your time range may say it is the latest data (may be 10 to 15
> min)
> > but you should be able to configure your memstore flushing in such a way
> > that there are no flushes happening for the latest data in that 10 to 15
> > min time.  Just saying my thoughts here.
> >
> >
> >
> >
> > On Mon, Jun 1, 2015 at 11:46 AM, Gautam Borah 
> > wrote:
> >
> > > Hi all,
> > >
> > > Here is our use case,
> > >
> > > We have a very write heavy cluster. Also we run periodic end point co
> > > processor based jobs that operate on the data written in the last 10-15
> > > mins, every 10 minute.
> > >
> > > Is there a way to only query in the MemStore from the end point
> > > co-processor? The periodic job scans for data using a time range. We
> > would
> > > like to implement a simple logic,
> > >
> > > a. if query time range is within MemStore's TimeRangeTracker, then
> query
> > > only memstore.
> > > b. If end Time of the query time range is within MemStore's
> > > TimeRangeTracker, but query start Time is outside MemStore's
> > > TimeRangeTracker (memstore flush happened), then query both MemStore
> and
> > > Files.
> > > c. If start time and end time of the query is outside of MemStore
> > > TimeRangeTracker we query only files.
> > >
> > > The incoming data is time series and we do not allow old data (out of
> > sync
> > > from clock) to come into the system(HBase).
> > >
> > > Cloudera has a scanner
> org.apache.hadoop.hbase.regionserver.InternalScan,
> > > that has methods like checkOnlyMemStore() and checkOnlyStoreFiles(). Is
> > > this available in Trunk?
> > >
> > > Also, how do I access the Memstore for a Column Family in the end point
> > > co-processor from CoprocessorEnvironment?
> > >
> >
>

Re: PhoenixIOException resolved only after compaction, is there a way to avoid it?

[Vladimir Rodionov]

>> Is IO exception is because of Phoenix
>> not able to read from multiple regions since error was resolved after the
>> compaction? or Any other thoughts?

Compaction does not decrease # of regions - it sorts/merges data into a
single file (in case of a major compaction) for every
region/column_family. SocketTimeout exception is probably because Phoenix
must read data from multiple files in
every region before compaction - this requires more CPU, more RAM and
produces more temp garbage.
Excessive GC activity, in turn, results in socket timeouts. Check GC logs
in RS and check RS logs for other errors -
they will probably give you a clue on what is going on during a query
execution.

-Vlad



On Mon, Jun 1, 2015 at 11:10 AM, Siva  wrote:

> Hi Everyone,
>
> We load the data to Hbase tables through BulkImports.
>
> If the data set is small, we can query the imported data from phoenix with
> no issues.
>
> If data size is huge (with respect to our cluster, we have very small
> cluster), I m encountering the following error
> (org.apache.phoenix.exception.PhoenixIOException).
>
> 0: jdbc:phoenix:172.31.45.176:2181:/hbase> selectcount(*)
> . . . . . . . . . . . . . . . . . . . . .>from  "ldll_compression"
>  ldll join "ds_compression"  ds on (ds."statusid" = ldll."statusid")
> . . . . . . . . . . . . . . . . . . . . .>where ldll."logdate"  >=
> '2015-02-04'
> . . . . . . . . . . . . . . . . . . . . .>and  ldll."logdate"  <=
> '2015-02-06'
> . . . . . . . . . . . . . . . . . . . . .>and ldll."dbname" =
> 'lmguaranteedrate';
> +--+
> | COUNT(1) |
> +--+
> java.lang.RuntimeException:
> org.apache.phoenix.exception.PhoenixIOException:
> org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36,
> exceptions:
> Mon Jun 01 13:50:57 EDT 2015, null, java.net.SocketTimeoutException:
> callTimeout=6, callDuration=62358: row '' on table 'ldll_compression'
> at
> region=ldll_compression,,1432851434288.1a8b511def7d0c9e69a5491c6330d715.,
> hostname=ip-172-31-32-181.us-west-2.compute.internal,60020,1432768597149,
> seqNum=16566
>
> at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2440)
> at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2074)
> at sqlline.SqlLine.print(SqlLine.java:1735)
> at sqlline.SqlLine$Commands.execute(SqlLine.java:3683)
> at sqlline.SqlLine$Commands.sql(SqlLine.java:3584)
> at sqlline.SqlLine.dispatch(SqlLine.java:821)
> at sqlline.SqlLine.begin(SqlLine.java:699)
> at sqlline.SqlLine.mainWithInputRedirection(SqlLine.java:441)
> at sqlline.SqlLine.main(SqlLine.java:424)
>
> I did the major compaction for "ldll_compression" through Hbase
> shell(major_compact 'ldll_compression'). Same query ran successfully after
> the compaction.
>
> 0: jdbc:phoenix:172.31.45.176:2181:/hbase> selectcount(*)
> . . . . . . . . . . . . . . . . . . . . .>from  "ldll_compression"
>  ldll join "ds_compression"  ds on (ds."statusid" = ldll."statusid")
> . . . . . . . . . . . . . . . . . . . . .>where ldll."logdate"  >=
> '2015-02-04'
> . . . . . . . . . . . . . . . . . . . . .>and  ldll."logdate"  <=
> '2015-02-06'
> . . . . . . . . . . . . . . . . . . . . .>and ldll."dbname" =
> 'lmguaranteedrate'
> . . . . . . . . . . . . . . . . . . . . .> ;
> +--+
> | COUNT(1) |
> +--+
> | 13480|
> +--+
> 1 row selected (72.36 seconds)
>
> Did anyone face the similar issue? Is IO exception is because of Phoenix
> not able to read from multiple regions since error was resolved after the
> compaction? or Any other thoughts?
>
> Thanks,
> Siva.
>

Re: Hbase vs Cassandra

[Vladimir Rodionov]

>> The key issue is that unless you need or want to use Hadoop, you
shouldn’t be using HBase. Its not a stand alone product or system.

Hello, what is use case of a big data application w/o Hadoop?

-Vlad

On Mon, Jun 1, 2015 at 2:26 PM, Michael Segel 
wrote:

> Saying Ambari rules is like saying that you like to drink MD 20/20 and
> calling it a fine wine.
>
> Sorry to all the Hortonworks guys but Amabari has a long way to go…. very
> immature.
>
> What that has to do with Cassandra vs HBase? I haven’t a clue.
>
> The key issue is that unless you need or want to use Hadoop, you shouldn’t
> be using HBase. Its not a stand alone product or system.
>
>
>
>
> > On May 30, 2015, at 7:40 AM, Serega Sheypak 
> wrote:
> >
> > 1. No killer features comparing to hbase
> > 2.terrible!!! Ambari/cloudera manager rulezzz. Netflix has its own tool
> for
> > Cassandra but it doesn't support vnodes.
> > 3. Rumors say it fast when it works;) the reason- it can silently drop
> data
> > you try to write.
> > 4. Timeseries is a nightmare. The easiest approach is just replicate data
> > to hdfs, partition it by hour/day and run spark/scalding/pig/hive/Impala
> >
> > пятница, 29 мая 2015 г. пользователь Ajay написал:
> >
> >> Hi,
> >>
> >> I need some info on Hbase vs Cassandra as a data store (in general plus
> >> specific to time series data).
> >>
> >> The comparison in the following helps:
> >> 1: features
> >> 2: deployment and monitoring
> >> 3: performance
> >> 4: anything else
> >>
> >> Thanks
> >> Ajay
> >>
>
>

Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs space

[Vladimir Rodionov]

>> Sounds like a possible
>> workaround would have been to bump the value
>> of  hfile.index.block.max.size, did you try that?

? How is that?

-Vlad

On Wed, Jun 3, 2015 at 11:03 AM, Esteban Gutierrez 
wrote:

> Thats a very interesting finding Sathya. Could you please open an HBase
> JIRA and additional information about this behavior? Sounds like a possible
> workaround would have been to bump the value
> of  hfile.index.block.max.size, did you try that?
>
> thanks,
> esteban.
>
> --
> Cloudera, Inc.
>
>
> On Tue, Jun 2, 2015 at 9:42 PM, sathyafmt  wrote:
>
> > FYI: haven't seen this issue since we turned off tsdb compaction. See
> > github
> > opentsdb issue #490 
> >  for
> > more info..
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4072091.html
> > Sent from the HBase User mailing list archive at Nabble.com.
> >
>

Re: How are version conflicts handled in HBase?

[Vladimir Rodionov]

Yes, last write wins (with higher sequenceId). MemStore will resolve this
conflict and only the last
put will be added eventually, unless ... between these two puts MemStore's
snapshot is created.
I this case put #1 will be saved in  a snapshot and eventually will make it
into a store file, but this is just my speculations.

-Vlad

On Thu, Jun 4, 2015 at 5:08 PM, Dia Kharrat  wrote:

> I'm trying to confirm the behavior of HBase when there are concurrent
> writes to the same cell that happen at the exact same millisecond and not
> providing a timestamp value to the Put operations (i.e. relying on current
> time of region server). Is it possible that such concurrent writes result
> in a cell with an identical version value or does HBase have a mechanism to
> protect against that?
>
> If that's the case, my understanding is that last write wins, correct?
>
> Thanks,
> Dia
>

Re: How are version conflicts handled in HBase?

[Vladimir Rodionov]

>> Please read http://hbase.apache.org/book.html#_store

How does this answer original question?

-Vlad

On Thu, Jun 4, 2015 at 6:30 PM, Ted Yu  wrote:

> Dia:
> Please read http://hbase.apache.org/book.html#_store
>
> Cheers
>
> On Thu, Jun 4, 2015 at 6:02 PM, Vladimir Rodionov 
> wrote:
>
> > Yes, last write wins (with higher sequenceId). MemStore will resolve this
> > conflict and only the last
> > put will be added eventually, unless ... between these two puts
> MemStore's
> > snapshot is created.
> > I this case put #1 will be saved in  a snapshot and eventually will make
> it
> > into a store file, but this is just my speculations.
> >
> > -Vlad
> >
> > On Thu, Jun 4, 2015 at 5:08 PM, Dia Kharrat  wrote:
> >
> > > I'm trying to confirm the behavior of HBase when there are concurrent
> > > writes to the same cell that happen at the exact same millisecond and
> not
> > > providing a timestamp value to the Put operations (i.e. relying on
> > current
> > > time of region server). Is it possible that such concurrent writes
> result
> > > in a cell with an identical version value or does HBase have a
> mechanism
> > to
> > > protect against that?
> > >
> > > If that's the case, my understanding is that last write wins, correct?
> > >
> > > Thanks,
> > > Dia
> > >
> >
>

Re: HConnection thread waiting on blocking queue indefinitely

[Vladimir Rodionov]

I would suggest reverting client config changes back to defaults. At least
we will know if the issue is somehow related to client config changes.
On Jun 5, 2015 6:15 AM, "ramkrishna vasudevan" <
ramkrishna.s.vasude...@gmail.com> wrote:

> Hbase:meta getting split? It may b some user region, can u check that? If
> ur meta was splitting then there is something wrong.
> Can u attach the log snippets.
>
> Sent from phone. Excuse typos.
> On Jun 5, 2015 6:00 PM, "mukund murrali"  wrote:
>
> > Hi
> >
> > In our case there at that instance when the client thread stalled, there
> > was a hbase:meta region split happening. So what went wrong? If there is
> a
> > split why should hconnection thread stall? Since we changed the client
> > configuration caused this? I am once again specifying our client related
> > changes we did
> >
> > hbase.client.retries.number => 5
> > zookeeper.recovery.retry => 0
> > zookeeper.session.timeout => 1000
> > zookeeper.recovery.retry.
> > intervalmilli => 1
> > hbase.rpc.timeout => 3.
> >
> > Is zk timeout too low?
> >
> >
> >
> >
> >
> >
> > On Fri, Jun 5, 2015 at 11:37 AM, ramkrishna vasudevan <
> > ramkrishna.s.vasude...@gmail.com> wrote:
> >
> > > When you started  your client server was the META table assigned.  May
> be
> > > some thing happened around that time and the client app was just
> waiting
> > on
> > > the meta table to be assigned.  It would have retried - Can you check
> the
> > > logs.?
> > >
> > > So the best part here is the stand alone client was able to be
> > successful -
> > > which means the new clients were able to talk successfully with the
> > > server.  And hence the restart of your client has solved  your problem.
> > It
> > > may be difficult to trouble shoot the exact issue with the limited
> info -
> > > but see if your client app regularly gets stalled and then it is better
> > to
> > > trouble shoot your app and the way it accesses the server.
> > >
> > > On Fri, Jun 5, 2015 at 11:21 AM, PRANEESH KUMAR <
> > praneesh.san...@gmail.com
> > > >
> > > wrote:
> > >
> > > > The client connection was in stalled state. But there was only one
> > > > hconnection thread found in our thread dump, which was waiting
> > > indefinitely
> > > > in BoundedCompletionService.take call. Meanwhile we ran a standalone
> > test
> > > > program which was successful.
> > > >
> > > > Once we restarted the client server, the problem got resolved.
> > > >
> > > > The basic doubt is, when the hconnection thread stalled, why the
> HBase
> > > > client failed to create any more hconnections(max pool size was 10).
> In
> > > > case of problem with table/meta regions how come the test program
> > > > succeeded.
> > > >
> > > > Regards,
> > > > Praneesh
> > > >
> > > > On Fri, Jun 5, 2015 at 10:21 AM, ramkrishna vasudevan <
> > > > ramkrishna.s.vasude...@gmail.com> wrote:
> > > >
> > > > > Can you tell us more. Is your client not working at all and it is
> > > > stalled ?
> > > > > Are you seeing some results but you find it slow than you expected?
> > > > >
> > > > > What type of workload are you running?  All the tables are healthy?
> > > Are
> > > > > you able to read or write to them individually using the hbase
> shell?
> > > > >
> > > > > On Fri, Jun 5, 2015 at 10:18 AM, PRANEESH KUMAR <
> > > > praneesh.san...@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Ram,
> > > > > >
> > > > > > The cluster ran without any problem for about 2 to 3 days with
> low
> > > > load,
> > > > > > once we enabled it for high load we immediately faced this issue.
> > > > > >
> > > > > >
> > > > > > Regards,
> > > > > > Praneesh.
> > > > > >
> > > > > > On Thursday 4 June 2015, ramkrishna vasudevan <
> > > > > > ramkrishna.s.vasude...@gmail.com> wrote:
> > > > > >
> > > > > > > Is your cluster in working condition.  Can you see if the META
> > has
> > > > been
> > > > > > > assigned properly?  If the META table is not initialized and
> > opened
> > > > > then
> > > > > > > your client thread will hang.
> > > > > > >
> > > > > > > Regards
> > > > > > > Ram
> > > > > > >
> > > > > > > On Thu, Jun 4, 2015 at 9:05 PM, PRANEESH KUMAR <
> > > > > > praneesh.san...@gmail.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > We are using Hbase-1.0.0. We also facing the same issue that
> > > client
> > > > > > > > connection thread is waiting at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1200).
> > > > > > > >
> > > > > > > > Any help is appreciated.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Praneesh
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: How are version conflicts handled in HBase?

[Vladimir Rodionov]

Agree. You should not rely on timestamp uniqueness to resolve version
conflicts. Design your key appropriately and provide support for conflict
resolution on a client side. Guide is a good example.
On Jun 5, 2015 8:58 AM, "Bryan Beaudreault" 
wrote:

> I wouldn't say it is recommended, but it is certainly possible to override
> the version timestamp at write time.  You might be able to use this to
> provide the uniqueness you need  (i.e. instead of using epoch timestamp,
> use one based more recently and add digits for uniqueness at the end).
>
> We've done this in the past and it worked perfectly fine, but i will say
> that eventually you usually want to take advantage of things like TTL, etc,
> and we ended up regretting the decision and migrating to a new table.
>
> Alternatively, just do uniqueness with the column qualifiers.  Your
> qualifier could be realQfBytes:guid, or something.  On the read side you
> can easily combine these together with a prefix filter.
>
> On Fri, Jun 5, 2015 at 11:55 AM Ted Yu  wrote:
>
> > Dia:
> > Have you tried command in the following form in hbase shell ?
> >
> >   hbase> t.get 'r1', {COLUMN => 'c1', TIMERANGE => [ts1, ts2], VERSIONS
> =>
> > 4}
> >
> > Cheers
> >
> > On Fri, Jun 5, 2015 at 8:50 AM, Dia Kharrat  wrote:
> >
> > > Ted: Yes, I've already read the HBase documentation, but didn't find
> > > anything that directly answered my question. My question was simply
> > whether
> > > it's possible to get cells with the same timestamp with concurrent Puts
> > to
> > > the same cell.
> > >
> > > Vlad: Thanks for the information. Yep, I've also noticed the behavior
> of
> > > the last-write-wins when testing directly in hbase shell and writing
> out
> > > two values to the same cell with an identical timestamp.
> > >
> > > So, I gather that an application writing to the same cell concurrently
> > will
> > > result in potential data loss when the write is at the exact same
> > > millisecond, correct? For my use-case, I'm not necessarily looking for
> > > uniqueness for the timestamp, but want to ensure I can still access all
> > > versions of the cell, even ones with the same timestamp.
> > >
> > > Dia
> > >
> > > On Thu, Jun 4, 2015 at 7:05 PM, Vladimir Rodionov <
> > vladrodio...@gmail.com>
> > > wrote:
> > >
> > > > >> Please read http://hbase.apache.org/book.html#_store
> > > >
> > > > How does this answer original question?
> > > >
> > > > -Vlad
> > > >
> > > > On Thu, Jun 4, 2015 at 6:30 PM, Ted Yu  wrote:
> > > >
> > > > > Dia:
> > > > > Please read http://hbase.apache.org/book.html#_store
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Thu, Jun 4, 2015 at 6:02 PM, Vladimir Rodionov <
> > > > vladrodio...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Yes, last write wins (with higher sequenceId). MemStore will
> > resolve
> > > > this
> > > > > > conflict and only the last
> > > > > > put will be added eventually, unless ... between these two puts
> > > > > MemStore's
> > > > > > snapshot is created.
> > > > > > I this case put #1 will be saved in  a snapshot and eventually
> will
> > > > make
> > > > > it
> > > > > > into a store file, but this is just my speculations.
> > > > > >
> > > > > > -Vlad
> > > > > >
> > > > > > On Thu, Jun 4, 2015 at 5:08 PM, Dia Kharrat 
> > > > wrote:
> > > > > >
> > > > > > > I'm trying to confirm the behavior of HBase when there are
> > > concurrent
> > > > > > > writes to the same cell that happen at the exact same
> millisecond
> > > and
> > > > > not
> > > > > > > providing a timestamp value to the Put operations (i.e. relying
> > on
> > > > > > current
> > > > > > > time of region server). Is it possible that such concurrent
> > writes
> > > > > result
> > > > > > > in a cell with an identical version value or does HBase have a
> > > > > mechanism
> > > > > > to
> > > > > > > protect against that?
> > > > > > >
> > > > > > > If that's the case, my understanding is that last write wins,
> > > > correct?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Dia
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: What cause hfile saved under /hbase/.archive besides snapshot?

[Vladimir Rodionov]

HBase moves old hfiles into archive directory after compaction is finished.
Compacted files are not deleted
because they can be still in use (some long running scanners) - that is why
they get moved into archive directory.
HBase can locate hfile in both: standard and archive directory. There is a
separate archive cleaner agent which runs
periodically and cleans archive from hfiles which are no longer used.

-Vlad

On Fri, Jun 5, 2015 at 12:25 PM, Tianying Chang  wrote:

> Hi,
>
> I found that there are many hfiles saved under /hbase/.archive/table/. I
> know when there is a  snapshot taken, the hfile might have to be saved
> under /hbase/.archive. But when we are not even taking snapshot, I saw lots
> of hfiles under /hbase/.archive. I noticed that if I run a major
> compaction, that seems clean up lots of hfiles from /hbase/.archive. But
> then it grows back again later.
>
> Wondering what else could cause hfiles stuck under /hbase/.archinve? Any
> hint is appreciated.
>
> Thanks
> Tian-YIng
>

Re: Query on OutOfOrderScannerNextException

[Vladimir Rodionov]

The scanner fails at the very beginning. The reason is because they need a
very few rows from a large file and HBase needs
to fill RPC buffer (which is 100 rows, yes?) before it can return first
batch. This takes more than 60 sec and scanner fails (do not ask me why its
not the timeout exception)

1. HBASE-13090 will help (can be back ported I presume to 1.0 and 0.98.x)
2. Smaller region size will help
3. Smaller  hbase.client.scanner.caching will help
4. Larger hbase.client.scanner.timeout.period will help
5. Better data store design (rowkeys) is preferred.

Too many options to choose from.

-Vlad


On Sat, Jun 6, 2015 at 3:04 PM, Arun Mishra  wrote:

> Thanks TED.
>
> Regards,
> Arun.
>
> > On Jun 6, 2015, at 2:34 PM, Ted Yu  wrote:
> >
> > HBASE-13090 'Progress heartbeats for long running scanners' solves the
> > problem you faced.
> >
> > It is in the 1.1.0 release.
> >
> > FYI
> >
> >> On Sat, Jun 6, 2015 at 12:54 PM, Arun Mishra  wrote:
> >>
> >> Hello,
> >>
> >> I have a query on OutOfOrderScannerNextException. I am using hbase
> 0.98.6
> >> with 45 nodes.
> >>
> >> I have a mapreduce job which scan 1 table for last 1 day worth data
> using
> >> timerange. It has been running fine for months without any failure. But
> >> last couple of days it has been failing with below exception. I have
> traced
> >> the failure to a single region. This region has 1 store and 1 hfile of
> >> 5+GB. What we realized was that, we were writing some bulk data, which
> used
> >> to land on this region. After we stopped writing this data, this region
> has
> >> been receiving very few writes per day.
> >>
> >> When mapreduce job runs, it creates a map task for this region and that
> >> task fails with OutOfOrderScannerNextException. I was able to reproduce
> >> this error by running a scan command with same start/stop row and
> timerange
> >> option. Finally, we split this region to be small enough for scan
> command
> >> to work.
> >>
> >> My query is if there is any option, apart from increasing the timeout,
> >> which can solve this use case? I am thinking of a use case where data
> comes
> >> in for 3 days a week in bulk and then nothing for next 3 days. Kind of
> >> creating a data hole in region.
> >> My understanding is that I am hit with this error because I have big
> store
> >> files and timerange scan is reading entire file even though it contains
> >> very few rowkeys for that timerange.
> >>
> >> hbase.client.scanner.caching = 100
> >> hbase.client.scanner.timeout.period = 60s
> >>
> >> scan 'dummytable',{ STARTROW=>'dummyrowkey-start',
> >> STOPROW=>'dummyrowkey-end', LIMIT=>1000,
> >> TIMERANGE=>[143346240,143354880]}
> >> ROW   COLUMN+CELL
> >>
> >> ERROR:
> org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:
> >> Expected nextCallSeq: 1 But the nextCallSeq got from client: 0;
> >> request=scanner_id: 33648 number_of_rows: 100 close_scanner: false
> >> next_call_seq: 0
> >> at
> >>
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3193)
> >> at
> >>
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29587)
> >> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
> >> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
> >> at
> >>
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
> >> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
> >> at java.lang.Thread.run(Thread.java:745)
> >>
> >>
> >> Regards,
> >> Arun
>

Re: Potential bugs in HTable In incrementColumnValue method

[Vladimir Rodionov]

Hi, Jerry

Which version of HBase is it?

-Vlad

On Tue, Jun 9, 2015 at 8:05 AM, Jerry Lam  wrote:

> Hi HBase community,
>
> Can anyone confirm that the method incrementColumnValue is implemented
> correctly?
>
> I'm talking about mainly the deprecated method:
>
>  @Deprecated
>
>   @Override
>
>   public long incrementColumnValue(final byte [] row, final byte [] family,
>
>   final byte [] qualifier, final long amount, final boolean writeToWAL)
>
>   throws IOException {
>
> return incrementColumnValue(row, family, qualifier, amount,
>
>   writeToWAL? Durability.SKIP_WAL: Durability.USE_DEFAULT);
>
>   }
>
>
> Note from the above, if writeToWAL is true, Durability is set to SKIP_WAL.
>
> It does not make sense to me so I'm asking if this might be a potential
> bug.
>
>
> Best Regards,
>
>
> Jerry
>

Potential issue with ExploringCompactionPolicy

[Vladimir Rodionov]

Repost from hbase-dev (for some reasons, it did not go through):

Hi, folks

This is from HBase book:
hbase.hstore.compaction.min.size
Description

A StoreFile smaller than this size will always be eligible for minor
compaction. HFiles this size or larger are evaluated by
hbase.hstore.compaction.ratio to determine if they are eligible. Because
this limit represents the "automatic include"limit for all StoreFiles
smaller than this value, this value may need to be reduced in write-heavy
environments where many StoreFiles in the 1-2 MB range are being flushed,
because every StoreFile will be targeted for compaction and the resulting
StoreFiles may still be under the minimum size and require further
compaction. If this parameter is lowered, the ratio check is triggered more
quickly. This addressed some issues seen in earlier versions of HBase but
changing this parameter is no longer necessary in most situations. Default:
128 MB expressed in bytes.
 Default

134217728
  hbase.hstore.compaction.max.size
Description

A StoreFile larger than this size will be excluded from compaction. The
effect of raising hbase.hstore.compaction.max.size is fewer, larger
StoreFiles that do not get compacted often. If you feel that compaction is
happening too often without much benefit, you can try raising this value.
Default: the value of LONG.MAX_VALUE, expressed in bytes.
 Default

9223372036854775807
 This is  applyCompactionPolicy (master branch):

 public List applyCompactionPolicy(final List
candidates,
   boolean mightBeStuck, boolean mayUseOffPeak, int minFiles, int
maxFiles) {

final double currentRatio = mayUseOffPeak
? comConf.getCompactionRatioOffPeak() :
comConf.getCompactionRatio();

// Start off choosing nothing.
List bestSelection = new ArrayList(0);
List smallest = mightBeStuck ? new ArrayList(0) :
null;
long bestSize = 0;
long smallestSize = Long.MAX_VALUE;

int opts = 0, optsInRatio = 0, bestStart = -1; // for debug logging
// Consider every starting place.
for (int start = 0; start < candidates.size(); start++) {
  // Consider every different sub list permutation in between start and
end with min files.
  for (int currentEnd = start + minFiles - 1;
  currentEnd < candidates.size(); currentEnd++) {
List potentialMatchFiles = candidates.subList(start,
currentEnd + 1);

// Sanity checks
if (potentialMatchFiles.size() < minFiles) {
  continue;
}
if (potentialMatchFiles.size() > maxFiles) {
  continue;
}

// Compute the total size of files that will
// have to be read if this set of files is compacted.
long size = getTotalStoreSize(potentialMatchFiles);

// Store the smallest set of files.  This stored set of files will
be used
// if it looks like the algorithm is stuck.
if (mightBeStuck && size < smallestSize) {
  smallest = potentialMatchFiles;
  smallestSize = size;
}

// BEGIN

if (size > comConf.getMaxCompactSize()) {
  continue;
}

++opts;
if (size >= comConf.getMinCompactSize()
&& !filesInRatio(potentialMatchFiles, currentRatio)) {
  continue;
}
// END
++optsInRatio;
if (isBetterSelection(bestSelection, bestSize, potentialMatchFiles,
size, mightBeStuck)) {
  bestSelection = potentialMatchFiles;
  bestSize = size;
  bestStart = start;
}
  }
}
if (bestSelection.size() == 0 && mightBeStuck) {
  LOG.debug("Exploring compaction algorithm has selected " +
smallest.size()
  + " files of size "+ smallestSize + " because the store might be
stuck");
  return new ArrayList(smallest);
}
LOG.debug("Exploring compaction algorithm has selected " +
bestSelection.size()
+ " files of size " + bestSize + " starting at candidate #" +
bestStart +
" after considering " + opts + " permutations with " + optsInRatio
+ " in ratio");
return new ArrayList(bestSelection);
  }


The code in question is between  // BEGIN //END
Why do we compare size of LIST of stores with configuration values which
are supposed to be per one store file?

Re: How to monitor and control heavy network traffic for region server

[Vladimir Rodionov]

Louis,

What do you mean by "monitor the long scan"? If you need to throttle
network IO during scan, you have to
do it on a client side. Take a look at
org.apache.hadoop.hbase.io.hadoopbackport.ThrottledInputStream
as an example, something similar you will need to implement on top of
ResultScanner - ThrottledResultScanner.

Good idea for improvement, actually.

-Vlad

On Wed, Jun 10, 2015 at 7:30 PM, Louis Hust  wrote:

> hi, Dave,
>
> For now we will not upgrade the version, so if there is something we can
> monitor the long scan for 0.96?
>
> 2015-06-11 2:00 GMT+08:00 Dave Latham :
>
> > I'm not aware of anything in version 0.96 that will limit the scan for
> > you - you may have to do it in your client yourself.  If you're
> > willing to upgrade, do check out the throttling available in HBase
> > 1.1:
> >
> https://blogs.apache.org/hbase/entry/the_hbase_request_throttling_feature
> >
> >
> >
> > On Wed, Jun 10, 2015 at 4:15 AM, 娄帅  wrote:
> > > any ideas?
> > >
> > > 2015-06-10 16:01 GMT+08:00 娄帅 :
> > >
> > >> Hi all,
> > >>
> > >> We are using HBASE 0.96 with Hadoop 2.2.0, recently we found there are
> > some
> > >> SCAN operations last for more than 1 hours, which lead to heavy
> network
> > >> traffic, because some data is not
> > >> stored at local data node and the region is very big, about 100G-500G。
> > >>
> > >> With heavy network traffic, the region server then can not offer
> service
> > >> cause the NIC is full.
> > >>
> > >> From DBA view point, I want to know if there is some operation for
> HBase
> > >> to limit the large scan or if there is some flow control for SCAN
> > operation
> > >> and how can we monitor the long SCAN operation。
> > >>
> >
>

Re: Stochastic Balancer by tables

[Vladimir Rodionov]

Balancer distributes regions among RS's - not equalizes them. You,
probably, have very serious data skew and not optimal key design. I would
suggest you to try custom region split policy and provide your own split
key, because standard splitting algorithm does not work well.
On Jun 18, 2015 7:50 AM, "Nasron Cheong"  wrote:

> Hi,
>
> I've noticed there are two settings available when using the HBase balancer
> (specifically the default stochastic balancer)
>
> hbase.master.balancer.stochastic.tableSkewCost
>
> hbase.master.loadbalance.bytable
>
> How do these two settings relate? The documentation indicates when using
> the stochastic balancer that 'bytable' should be set to false?
>
> Our deployment relies on very few, very large tables, and I've noticed bad
> distribution when accessing some of the tables. E.g. there are 443 regions
> for a single table, but when doing a MR job over a full scan of the table,
> the first 426 regions scan quickly (minutes), but the remaining 17 regions
> take significantly longer (hours)
>
> My expectation is to have the balancer equalize the size of the regions for
> each table.
>
> Thanks!
>
> - Nasron
>

Re: OutOfOrderScannerNextException consistently killing mapreduce jobs

[Vladimir Rodionov]

Mateusz,

- How many regions do you have in your table?
- What is the cluster size?
- What is the scan spec in your M/R job (time range, filters)
- RS node spec (CPUs, RAM, disks)

Ted's link is a good start point.

-Vlad

On Thu, Jun 25, 2015 at 10:19 AM, Ted Yu  wrote:

> Have you read this thread http://search-hadoop.com/m/YGbb1sOLh2W9Z9z ?
>
> Cheers
>
> On Thu, Jun 25, 2015 at 10:10 AM, Mateusz Kaczynski 
> wrote:
>
> > One of our clusters running HBase 0.98.6-cdh5.3.0 used to work
> (relatively)
> > smoothly until a couple of days ago, when out of the sudden jobs stated
> > grinding to a halt and getting killed upon reporting a massive amount of
> > errors of form:
> >
> > org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of
> > OutOfOrderScannerNextException: was there a rpc timeout?
> > at
> > org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:410)
> > at
> >
> >
> org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:230)
> > at
> >
> >
> org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:138)
> > at
> >
> >
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:483)
> > at
> >
> >
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:76)
> > at
> >
> >
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:85)
> > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:139)
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
> > at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:415)
> > at
> >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
> > at org.apache.hadoop.mapred.Child.main(Child.java:262)
> >
> > HBase regionservers contain a bunch of:
> > WARN  [B.defaultRpcServer.handler=16,queue=1,port=60020] ipc.RpcServer:
> > B.defaultRpcServer.handler=16,queue=1,port=60020: caught a
> > ClosedChannelException, this means that the server was processing a
> request
> > but the client went away. The error message was: null
> >
> > and:
> > INFO  [regionserver60020.leaseChecker] regionserver.HRegionServer:
> Scanner
> > 1086 lease expired on region
> >
> >
> table,8bf8fc3cd0e842c00fb4e556bbbdcd0f,1420155383100.19f5ed7c735d33b2cf8997e0b373a1a7
> >
> > in addition there are reports of compactions (not sure if relevant at
> all):
> > regionserver.HStore: Completed major compaction of 3 file(s) in cf of
> >
> >
> table,fc0caf49fa871a61702fa3781e160101,1420728621152.9ccc317ca180cabde13864d4600c8693.
> > into efd8bec4dbf54ccca5f1351bfe9890c3(size=5.9 G), total size for store
> is
> > 5.9 G. This selection was in queue for 0sec, and took 1mins, 57sec to
> > execute.
> >
> > I've adjusted the following, thinking it might be scanner cache size
> issue
> > (we're dealing with docs of circa 100kb):
> > hbase.rpc.timeout - 90
> > hbase.regionserver.lease.period - 45
> > hbase.client.scanner.timeout.period - 45
> > hbase.client.scanner.caching - (down to) 50
> >
> > To no avail. So I stripped the hbase config from hbase-site.xml to bare
> > minumum but I can reproduce it with a striking accuracy. The minimalistic
> > job reads from a table(c 3500 regions, 17 nodes), uses NullOutputFormat
> but
> > doesn't write to it, mappers's map function doesn't do anything.
> >
> > It starts pretty fast getting through 1.75% of the specified scan in ~1
> > minute. Then hits 2.5% in ~2m, 3% in ~3m. Then around 4m20s, a massive
> wave
> > of aforementioned OutOfOrderScannerNextException starts pouring in,
> slowing
> > the job down until it fails ~1h later.
> >
> > I checked the nodes memory and disk usage on the individual nodes - all
> > good, open file permissions are set relatively high, we're clearly not
> > hitting the limit.
> >
> > I'm running out of sanity and was wondering if anyone might have any
> ideas?
> >
> >
> > --
> > *Mateusz Kaczynski*
> >
>

Re: Could not reseek StoreFileScanner

[Vladimir Rodionov]

Check your NameNode/DataNode log files. There will be some additional bits
of information there.

-Vlad

On Sat, Jun 27, 2015 at 6:14 AM, Ted Yu  wrote:

> Please provide a bit more information:
>
> the hbase / hadoop release you use
> the type of data block encoding for the table
>
> How often did this happen ?
>
> thanks
>
>
> On Sat, Jun 27, 2015 at 3:44 AM, غلامرضا  wrote:
>
> > hi
> >
> > i got this  exception in reduce task when task try to incement table.
> >
> > Jun 27 12:42:56 10.3.72.94 [INFO]-2015/06/27
> > 12:42:56-AsyncProcess.logAndResubmit(713) - #10486, table=table1,
> > attempt=10/35 failed 4 ops, last exception: java.io.IOException:
> > java.io.IOException: Could not reseek StoreFileScanner[HFileScanner for
> > reader
> >
> reader=hdfs://m2/hbase2/data/default/table1/d52beedee15de2e7bb380f14bb0929fb/c2/daa0269a1f1c44f3811a25976b9278c8_SeqId_95_,
> > compression=snappy, cacheConf=CacheConfig:enabled [cacheDataOnRead=true]
> > [cacheDataOnWrite=false] [cacheIndexesOnWrite=false]
> > [cacheBloomsOnWrite=false] [cacheEvictOnClose=false]
> > [cacheDataCompressed=false] [prefetchOnOpen=false], ...
> > Jun 27 12:42:56 10.3.72.94
> > ...firstKey=\x00KEY1\x013yQ/c2:\x03\x00\x03^D\xA9\xC4/1435136203460/Put,
> > lastKey=\x00KEYN\x013yS/c2:\x03\x00\x02\xAE~A\xE0/1435136896864/Put,
> > avgKeyLen=36, avgValueLen=68, entries=15350817, length=466678923,
> > cur=\x00KEY2\x013yT/c2:/OLDEST_TIMESTAMP/Minimum/vlen=0/mvcc=0] to key
> > \x00KEY3\x013yT/c2:\x00fhamrah/LATEST_TIMESTAMP/Maximum/vlen=0/mvcc=0
> > at
> >
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:184)
> > at
> >
> org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:55)
> > at org...
> > Jun 27 12:42:56 10.3.72.94
> >
> apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:313)
> > at
> >
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:269)
> > at
> >
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:741)
> > at
> >
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekAsDirection(StoreScanner.java:729)
> > at
> >
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:546)
> > at
> >
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:140)
> > at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerIm...
> > Jun 27 12:42:56 10.3.72.94 ...pl.populateResult(HRegion.java:4103)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:4183)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:4061)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:4030)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:4017)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:5010)
> > at
> > org.apache.hadoop.hbase.regionserver.HRegion.increment(HRegion.java:5611)
> > at org.apache.hado...
> > Jun 27 12:42:56 10.3.72.94
> > ...op.hbase.regionserver.HRegionServer.increment(HRegionServer.java:4452)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3673)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3607)
> > at
> >
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:30954)
> > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093)
> > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> > at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcE...
> > Jun 27 12:42:56 10.3.72.94 ...xecutor.java:130)
> > at
> org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.io.IOException: Failed to read compressed block at
> > 302546975, onDiskSizeWithoutHeader=67898, preReadHeaderSize=33,
> > header.length=33, header bytes:
> >
> DATABLK*\x00\x00D\x16\x00\x01\x00Q\x00\x00\x00\x00\x01}\x1D\x98\x01\x00\x00@
> > \x00\x00\x00D/
> > at
> >
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1549)
> > at
> >
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:141...
> > Jun 27 12:42:56 10.3.72.94 ...3)
> > at
> >
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:394)
> > at
> >
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:253)
> > at
> >
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:539)
> > at
> >
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.rese

Re: Scan got exception

[Vladimir Rodionov]

I believe CDH 5.2.0 does not contain all BucketCache critical patches, but
I may be wrong.

-Vlad

On Tue, Jun 30, 2015 at 12:25 AM, Louis Hust  wrote:

> 
> hbase.bucketcache.size
> 80
> hbase-site.xml
> 
> 
> hbase.bucketcache.ioengine
> file:/export/hbase/cache.data
> hbase-site.xml
> 
> 
> hbase.bucketcache.combinedcache.enabled
> false
> hbase-site.xml
> 
>
> 2015-06-30 12:22 GMT+08:00 Ted Yu :
>
> > How do you configure BucketCache ?
> >
> > Thanks
> >
> > On Mon, Jun 29, 2015 at 8:35 PM, Louis Hust 
> wrote:
> >
> > > BTW, the hbase is hbase0.98.6 CHD5.2.0
> > >
> > > 2015-06-30 11:31 GMT+08:00 Louis Hust :
> > >
> > > > Hi, all
> > > >
> > > > When I scan a table using hbase shell, got the following message:
> > > >
> > > > hbase(main):001:0> scan 'atpco:ttf_record6'
> > > > ROW  COLUMN+CELL
> > > >
> > > > ERROR:
> > org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException:
> > > > Expected nextCallSeq: 1 But the nextCallSeq got from client: 0;
> > > > request=scanner_id: 201542113 number_of_rows: 100 close_scanner:
> false
> > > > next_call_seq: 0
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3193)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29587)
> > > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
> > > > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
> > > > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
> > > > at java.lang.Thread.run(Thread.java:744)
> > > >
> > > >
> > > > *And the region server got the following error:*
> > > >
> > > > 2015-06-30 11:08:11,877 ERROR
> > > > [B.defaultRpcServer.handler=27,queue=0,port=60020] ipc.RpcServer:
> > > > Unexpected throwable object
> > > > java.lang.IllegalArgumentException: Negative position
> > > > at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:675)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.io.hfile.bucket.FileIOEngine.read(FileIOEngine.java:87)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getBlock(BucketCache.java:406)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.io.hfile.LruBlockCache.getBlock(LruBlockCache.java:389)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:635)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:749)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:136)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:108)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:507)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:140)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3900)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3980)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3858)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3849)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3245)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29587)
> > > > at
> > > org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
> > > > at
> > > org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
> > > >
> > >
> >
>