This is my HBASE GC options of CMS, it does work well.
XX:+DisableExplicitGC -XX:+UseCompressedOops -XX:PermSize=160m
-XX:MaxPermSize=160m -XX:GCTimeRatio=19 -XX:SoftRefLRUPolicyMSPerMB=0
-XX:SurvivorRatio=2 -XX:MaxTenuringThreshold=1 -XX:+UseFastAccessorMethods
-XX:+UseParNewGC -XX:+UseConcMarkSw
Hi,
Check http://blog.sematext.com/2013/06/24/g1-cms-java-garbage-collector/
Those graphs show RegionServer before and after switch to G1. The
dashboard screenshot further below shows CMS (top row) vs. G1 (bottom
row). After those tests we ended up switching to G1 across the whole
cluster and h
thanks for all these valuable comments.
Jason
On Mon, Jul 8, 2013 at 12:25 PM, Michael Segel wrote:
> Where is murmur?
>
> In your app?
> So then every app that wants to fetch that row must now use murmur.
>
> Added to Hadoop/HBase?
> Then when you do upgrades you have to make sure that the pack
We haven't disable block cache. So I doubt that's the problem.
On Mon, Jul 8, 2013 at 4:50 PM, Varun Sharma wrote:
> FYI, if u disable your block cache - you will ask for "Index" blocks for
> every single request. So such a high rate of request is plausible for Index
> blocks even when your requ
meta blocks are at the end:
http://hbase.apache.org/book.html#d2617e12979, a way to tell would be
by logging from the HBase side but then I guess it's hard to reconcile
with which file we're actually reading from...
Regarding your second question, you are asking if we block HDFS
blocks? We don't,
FYI, if u disable your block cache - you will ask for "Index" blocks for
every single request. So such a high rate of request is plausible for Index
blocks even when your requests are totally random on your data.
Varun
On Mon, Jul 8, 2013 at 4:45 PM, Viral Bajaria wrote:
> Good question. When I
Good question. When I looked at the logs, it's not clear from it whether
it's reading a meta or data block. Is there any kind of log line that
indicates that ? Given that it's saying that it's ready from a startOffset
I would assume this is a data block.
A question that comes to mind, is this read
Do you know if it's a data or meta block?
J-D
On Mon, Jul 8, 2013 at 4:28 PM, Viral Bajaria wrote:
> I was able to reproduce the same regionserver asking for the same local
> block over 300 times within the same 2 minute window by running one of my
> heavy workloads.
>
> Let me try and gather so
I was able to reproduce the same regionserver asking for the same local
block over 300 times within the same 2 minute window by running one of my
heavy workloads.
Let me try and gather some stack dumps. I agree that jstack crashing the
jvm is concerning but there is nothing in the errors to know w
On Mon, Jul 8, 2013 at 12:22 PM, Viral Bajaria wrote:
> - I tried taking a stack trace using jstack but after the dump it crashed
> the regionserver. I also did not take the dump on the offending
> regionserver, rather took it on the regionservers that were making the
> block count. I will take a
Thanks guys for going through that never-ending email! I will create the
JIRA for block cache eviction and the regionserver assignment command. Ted
already pointed to the JIRA which tries to go a different datanode if the
primary is busy (I will add comments to that one).
To answer Andrews' questi
On Mon, Jul 8, 2013 at 11:09 AM, Suraj Varma wrote:
> Hello:
> We have an HBase cluster with region servers running on 8GB heap size with
> a 0.6 block cache (it is a read heavy cluster, with bursty write traffic
> via MR jobs). (version: hbase-0.94.6.1)
>
> During HBaseCon, while speaking to a f
@Vladimir: Yes - 8GB heap pauses are quite low and around what you mention.
My concern was about the pauses that *may* be triggered when I move to 24GB
(i.e. much larger heaps), for instance.
@Kevin: Thanks! I think mslab is already enabled by default now.
CMSParallelRemarkEnabled is interesting -
>>I'm especially concerned about long pauses causing zk session timeouts and
>>consequent RS shutdowns. Our boxes do have a lot of RAM and we are
>>exploring how we can use more of it for the cluster while maintaining
>>overall stability.
Suraj, what is the maximum GC pause you have observed in yo
Hey Suraj,
I would recommend turning on MSlab and using the following settings:
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled
On Mon, Jul 8, 2013 at 2:09 PM, Suraj Varma wrote:
> Hello:
> We have an HBase cluster with region se
Hello:
We have an HBase cluster with region servers running on 8GB heap size with
a 0.6 block cache (it is a read heavy cluster, with bursty write traffic
via MR jobs). (version: hbase-0.94.6.1)
During HBaseCon, while speaking to a few attendees, I heard some folks were
running region servers as h
Viral,
>From what you described here I can conclude that either:
1. your data access pattern is random and uniform (no data locality at all)
2. or you trash block cache with scan operations with block cache enabled.
or both.
but the idea of treating local and remote blocks differently is very g
Boris if your working set does not fit comfortably into 0.4 * 9 * 12GB ~ 43GB
HBase cluster block cache ( and I suppose
it does not, because it is less than 1% of your data set ) - you will be
depended on your local data nodes (OS page cache + raw disk I/O) performance.
Your maximum raw I/O is
Hello Michael,
looking in the code, it seems to me that the 60s is hardcoded, however it
retries for, on default, 10 times, so in total 10 minutes wait time, I upped
that to 20 times, so now it is 20 minutes for me, but still, we have some
pretty big regions whose compaction (which was the case
Where is murmur?
In your app?
So then every app that wants to fetch that row must now use murmur.
Added to Hadoop/HBase?
Then when you do upgrades you have to make sure that the package is still in
your class path. Note that different vendor's release management will mean YMMV
as to what ha
I just don't understand.. every creation/interpretation of the key is
going to require code to be used. The murmur implementation is with
that code. How is there any extra burden?
On Mon, Jul 8, 2013 at 11:54 AM, Michael Segel
wrote:
> You will need to put the jar into either every app that runs,
http://jsperf.com/murmur3-performance might be related.
On Mon, Jul 8, 2013 at 8:54 AM, Michael Segel wrote:
> You will need to put the jar into either every app that runs, or you will
> need to put it on every node. Every upgrade, you will need to make sure its
> still in your class path.
>
> Mo
You will need to put the jar into either every app that runs, or you will need
to put it on every node. Every upgrade, you will need to make sure its still in
your class path.
More work for the admins. So how much faster is it over MD5? MD5 and SHA-1 are
part of the Java libraries that ship w
> Would it make sense to give remote blocks higher priority over the local
blocks that can be read via SCR and not let them get evicted if there is a
tie in which block to evict ?
That sounds like a reasonable idea. As are the others.
But first, could this be a bug?
What version of HBase? Were
In 0.94, we have src/main/java/org/apache/hadoop/hbase/util/MurmurHash.java
For hadoop 1, there is src/core/org/apache/hadoop/util/hash/MurmurHash.java
Cheers
On Mon, Jul 8, 2013 at 8:29 AM, Michael Segel wrote:
> Is murmur part of the standard java libraries?
>
> If not, you end up having to d
On Mon, Jul 8, 2013 at 11:29 AM, Michael Segel
wrote:
> If not, you end up having to do a bit more maintenance of your cluster and
> that's going to be part of your tradeoff.
How so?
-Mike
Is murmur part of the standard java libraries?
If not, you end up having to do a bit more maintenance of your cluster and
that's going to be part of your tradeoff.
On Jul 8, 2013, at 10:14 AM, Mike Axiak wrote:
> Hello Jason,
>
> Have you considered the following rowkey?
>
> murmur_128(us
Ian,
You still want to stick to your relational modeling. :-(
You need to play around more with hierarchical models to get a better
appreciation.
If you model as if you're working with a RDBMS then you will end up with a poor
HBase table design.
In ERD models, you don't have the concept o
Not saying this is a solution or better in anyway but just more food for
thought. Is there any maximum size limit for UserIds? You can pad also for
Users Ids of smaller length. You are using more space in this way though.
It can help in sorting as well.
Regards,
Shahab
On Mon, Jul 8, 2013 at 10:
Hello Jason,
Have you considered the following rowkey?
murmur_128(userId) + timestamp + userId ?
This handles both of your cases as (1) murmur 128 is much faster than
md5 so will have very low overhead and (2) the userid at the end of
the key will ensure that no murmur collisions will cause is
Hello Michael,
looking in the code, it seems to me that the 60s is hardcoded, however
it retries for, on default, 10 times, so in total 10 minutes wait
time, I upped that to 20 times, so now it is 20 minutes for me, but
still, we have some pretty big regions whose compaction (which was the
case in
Hello,
I am trying to get some advice on pros/cons of using separator/delimiter as
part of HBase row key.
Currently one of our user activity tables has a rowkey design of
"UserID^TimeStamp" with a separator of "^". (UserID is a string that won't
include '^').
This is designed for the two common
For suggestion #3 below, take a look at:
HBASE-7509 Enable RS to query a secondary datanode in parallel, if the
primary takes too long
Cheers
On Mon, Jul 8, 2013 at 3:04 AM, Viral Bajaria wrote:
> Hi,
>
> TL;DR;
> Trying to make a case for making the block eviction strategy smart and to
> not e
Moving to HBase user mailing list.
Can you upgrade to newer release such as 0.94.8 ?
Cheers
On Jul 8, 2013, at 4:36 AM, Boris Emelyanov wrote:
> I'm trying to configure hbase for fully random read performance, my cluster
> parameters are:
>
> 9 servers as slaves, each has two 1TB HDD as had
Hi,
TL;DR;
Trying to make a case for making the block eviction strategy smart and to
not evict remote blocks more frequently and make the requests more smarter.
The question here comes after I debugged the issue that I was having with
random region servers hitting high load averages. I initially
This issue is not yet solved.
I am using hadoop-core-1.0.4.jar with hbase-0.94.8.jar .
thanks in advance for any help.
--
View this message in context:
http://apache-hbase.679495.n3.nabble.com/java-lang-RuntimeException-java-lang-ClassNotFoundException-org-apache-hadoop-hbase-ipc-WritableRp
36 matches
Mail list logo
