Hi Vladimir!
Here are graphs. Servlet (3 tomcats on 3 different hosts write to HBase)
http://www.bigdatapath.com/wp-content/uploads/2015/08/01_apps1.png
See how response time jump. I can't explain it. Write load is really-really
low.
all RS have even load. I see request-metrics in HBase master
How about GC activity? ApplicationStopTime? Do you track that?
Is the issue reproducible? or you got it first time?
Start with RS logs and try to find anything suspicious in a period of a
very high latency. 1.5 sec HBase write latency does not look right.
-Vlad
On Tue, Aug 11, 2015 at 2:08 PM,
How about GC activity? ApplicationStopTime? Do you track that?
yes, jviusalm says it's ok, newrelic also doesn't show something strange.
HBase also says it's OK.
Profiler says most time thread is waiting for response from hbase side. My
assumption is:
1. I have weird bug in HBase configuration
2.
Probably I found something. Response time decreases when parallelism grows.
What I did:
1. wrap business logic controller into java main class. My controller does
some logic and puts/gets to hbase with checkAndPut (sometimes)
2. create HConnection
3. pass HConnection to controller
4. wrap
We are using hbase-1.0.0. The following logs appears for all regions in the
regionserver
2015-08-08 14:01:51,586 DEBUG [regionserver//R1:16020.compactionChecker]
compactions.RatioBasedCompactionPolicy: Skipping major compaction of
bq. oldestTime -9223370597787064221ms
This is due to minTimestamp missing from store file:
Long minTimestamp = sf.getMinimumTimestamp();
long oldest = (minTimestamp == null)
? Long.MIN_VALUE
: now - minTimestamp.longValue();
Can you pastebin log from
Hi, we are using version 1.0.0+cdh5.4.4+160
We have heavy write load, ~ 10K per econd
We have 10 nodes 7 disks each. I read some perf notes, they state that
HBase can handle 1K per second writes per node without any problems.
I see some spikes on writers. Write operation timing jumps from
Yes. We have min version to be 1, hence I suppose there wont be any sort of
time based compaction. After triggering major compaction we had only the
following logs
compactions.RatioBasedCompactionPolicy: Selecting compaction from 3 store
files, 0 compacting, 3 eligible, 50 blocking
Thanks
On
*Common questions:*
1. How large is your single write?
2. Do you see any RegionTooBusyException in a client log files
3. How large is your table ( # of regions, # of column families)
4. RS memory related config: Max heap
5. memstore size (if not default - 0.4)
Memstore flush
Monitor GC events (application stop time). Your RS may have nonoptimal
hotspot GC settings. Search Internet on how to tune GC large heaps.
-Vlad
On Tue, Aug 11, 2015 at 9:54 AM, Vladimir Rodionov vladrodio...@gmail.com
wrote:
*Common questions:*
1. How large is your single write?
2.
Can you post code snippet? Pastbin link is fine.
-Vlad
On Tue, Aug 11, 2015 at 4:03 PM, Serega Sheypak serega.shey...@gmail.com
wrote:
Probably I found something. Response time decreases when parallelism grows.
What I did:
1. wrap business logic controller into java main class. My
11 matches
Mail list logo
