Hello,
Recently our cluster which has been running fine for 2 weeks split to 1024
regions at 1GB per region, after this split the cluster is unusable. Using the
performance benchmark I was getting a little better than 100 w/s, whereas
before it was 5000 w/s. There are 15 nodes of m2.2xlarge wit
Can you provide a bit more information (such as HBase release) ?
If you pastebin one of the region servers' log, that would help us
determine the cause.
Cheers
On Wed, Nov 5, 2014 at 9:29 PM, Pere Kyle wrote:
> Hello,
>
> Recently our cluster which has been running fine for 2 weeks split to 1
IncreasingToUpperBoundRegionSplitPolicy is the default split policy.
You can read the javadoc of this class to see how it works.
Cheers
On Wed, Nov 5, 2014 at 9:39 PM, Ted Yu wrote:
> Can you provide a bit more information (such as HBase release) ?
>
> If you pastebin one of the region servers
Here is a paste from one of the region servers
http://pastebin.com/H3BaHdPq
I am running Hbase on EMR
HBase Version 0.94.18, re17f91a1f107923d2defc7f18dbca59983f0a69f
Thanks,
Pere
On Nov 5, 2014, at 9:39 PM, Ted Yu wrote:
> Can you provide a bit more information (such as HBase release) ?
>
Ted,
Thanks so much for that information. I now see why this split too often, but
what I am not sure of is how to fix this without blowing away the cluster. Add
more heap?
Another symptom I have noticed is that load on the Master instance hbase daemon
has been pretty high (load average 4.0, wh
You can use ConstantSizeRegionSplitPolicy.
Split policy can be specified per table. See the following example
in create.rb :
hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO =>
'HexStringSplit'}
In 0.94.18, there isn't online merge. So you have to use other method to
merge the small regio
Watching closely a region server in action. It seems that the memstores are
being flushed at around 2MB on the regions. This would seem to indicate that
there is not enough heap for the memstore and I am hitting the upper bound of
limit (default). Would this be a fair assumption? Should I look
bq. to increase heap and increase the memstore limit?
Yes. That would be an action that bears fruit.
Long term, you should merge the small regions.
Cheers
On Wed, Nov 5, 2014 at 11:20 PM, Pere Kyle wrote:
> Watching closely a region server in action. It seems that the memstores
> are being flu
So I have set the heap to 12Gb and the memstore limit to upperLimit .5
lowerLimit .45. I am not seeing any changes in behavior from the cluster so
far, i have restarted 4/17 region servers. Do I need to restart master for the
memstore to take effect? Also how do I enable logging to show why a re
bq. Do I need to restart master for the memstore to take effect?
No. memstore is used by region server.
Looks like debug logging was not turned on (judging from your previous
pastebin).
Some of flush related logs are at INFO level. e.g. Do you see any of the
following log ?
LOG.info("Flush
Thanks again for your help!
I do not see a single entry in my logs for memstore pressure/global heap. I do
see tons of logs from the periodicFlusher:
http://pastebin.com/8ZyVz8AH
This seems odd to me. Today alone there are 1829 flushes from periodicFlusher.
Is there some other log4j property I
The default periodic flush is 1 hour. If you have a lot of regions and your
write distribution is not strictly uniform this can cause a lot of small
flushes, as you are seeing. I tuned this up to 12 hours in my cluster, and
may tune it up further. It made a big impact on the number of minor
compa
So set this property?
hbase.regionserver.optionalcacheflushinterval
4320
hbase-default.xml
Do I need to set this as well?
hbase.regionserver.logroll.period
360
hbase-default.xml
Thanks,
Pere
On Nov 6, 2014, at 11:23 AM, Bryan Beaudreault wrote:
> The default periodic flush is
The problem is regions dont get uniform writes, but HLogs are on a
regionserver level (at least in hbase 0.94), so that shouldn't be a
problem. I would keep it how it is.
Also this may not fix everything but will reduce compaction load. You can
also up hbase.hstore.compaction.max to compact more
Bryan,
Thanks so much for the in depth details. The workload I am trying to account
for is write heavy/scan heavy. This is an analytics cluster that will need
about 500-5000 w/s and handle map reduce jobs. I see your recommendation of the
i2.2xl, would you recommend this over the m2.4xlarge? Th
The i2 class instance is in the latest generation of instance in EC2 so has
newer everything, and has SSD. That's the biggest difference between
m2.4xlarge. HBase is by no means optimized for SSD but we did notice that
these perform MUCH better than the traditional disks in EC2. Our workload
is
Bryan,
Thanks again for the incredibly useful reply.
I have confirmed that the callQueueLen is in fact 0, with a max value of 2 in
the last week (in ganglia)
hbase.hstore.compaction.max was set to 15 on the nodes, from a previous 7.
Freezes (laggy responses) on the cluster are frequent and af
Ive been doing some testing with ITMTTR recently in ec2 with m1.l and m1.xl
instances. Debug level logging seems to produce 20-30 messages/sec on the
RS. I have noticed pauses in the log entries that last anywhere from 30-120
seconds. I have no explanation for the pauses other than the environment.
Ive been doing some testing with ITMTTR recently in ec2 with m1.l and m1.xl
instances. Debug level logging seems to produce 20-30 messages/sec on the
RS. I have noticed pauses in the log entries that last anywhere from 30-120
seconds. I have no explanation for the pauses other than the environment.
One other thought: you might try tracing your requests to see where the
slowness happens. Recent versions of PerformanceEvaluation support this
feature and can be used directly or as an example for adding tracing to
your application.
On Thursday, November 6, 2014, Pere Kyle wrote:
> Bryan,
>
> T
So I have another symptom which is quite odd. When trying to take a snapshot of
the the table with no writes coming in (i stopped thrift) it continually times
out when trying to flush (i don’t believe i have the option of non flush in
.94). Every single time I get:
ERROR: org.apache.hadoop.hbas
Writes stay in the memstore until the memstore limit is hit or a flush is
otherwise invoked (periodic, etc). So it makes sense there is still a lot
to be flushed when you do the snapshot.
To me this points to a disk write throughput issue, which makes sense since
you only have 1 of them.
During t
First of all I would like to thank you all for your help on this issue,
especially Ted, Bryan, and Nick.
It turned out the issue was the GMOND daemon.
When the regions split more hbase metrics were sent to the ganglia daemon. This
resulted in ganglia saturating the I/O on the main ephemeral st
23 matches
Mail list logo
