Are your GC logs enabled? Can you see any long pause in it?
On Fri, Aug 30, 2013 at 4:45 AM, Kiru Pakkirisamy <kirupakkiris...@yahoo.com > wrote: > I just moved from 0.94.10 to 0.94.11. Tremendous improvement in our app's > query response. Went down to 1.3 sec from 1.7 sec. > Concurrent tests are also good, but it still exponentially degrades from > to 10 secs for 8 concurrent clients. There might a bug lurking in there > somewhere that is probably affecting us. > > > Regards, > - kiru > > ________________________________ > From: Federico Gaule <fga...@despegar.com> > To: user@hbase.apache.org > Sent: Thursday, August 29, 2013 5:37 AM > Subject: Re: experiencing high latency for few reads in HBase > > > In 0.94.11 Release, has been included an optimization for MultiGets: > https://issues.apache.org/jira/browse/HBASE-9087 > > What version have you deployed? > > > On 08/29/2013 01:29 AM, lars hofhansl wrote: > > A 1s SLA is tough in HBase (or any large memory JVM application). > > > > > > Maybe, if you presplit your table, play with JDK7 and the G1 collector, > but nobody here will vouch for such an SLA in the 99th percentile. > > I heard some folks have experimented with 30GB heaps and G1 and have > reported max GC times of 200ms, but I have not verified that. > > > > -- Lars > > > > > > > > ----- Original Message ----- > > From: Saurabh Yahoo <saurabh...@yahoo.com> > > To: "user@hbase.apache.org" <user@hbase.apache.org> > > Cc: "user@hbase.apache.org" <user@hbase.apache.org> > > Sent: Wednesday, August 28, 2013 3:17 PM > > Subject: Re: experiencing high latency for few reads in HBase > > > > Hi Vlad, > > > > Thanks for your response. > > > > 1. Our SLA is less than one sec. we cannot afford latency more than 1 > sec. > > > > We can increase heap size if that help, we have enough memory on server. > What would be the optimal heap size? > > > > 2. Cache hit ratio is 95%. One thing I don't understand that we have > allocated only 4gb for block cache out of 12gb. That left 8gb for rest of > JVM. There is no write. Memcache is empty. Is 8gb not enough for hbase to > process the requests? What are the most memory consuming objects in region > server? > > > > 3. We will change the cf to IN_memory and report back performance > difference. > > > > Thanks, > > Saurabh. > > > > On Aug 28, 2013, at 3:15 PM, Vladimir Rodionov <vrodio...@carrieriq.com> > wrote: > > > >> 1. 4 sec max latency is not that bad taking into account 12GB heap. It > can be much larger. What is your SLA? > >> 2. Block evictions is the result of a poor cache hit rate and the root > cause of a periodic stop-the-world GC pauses (max latencies > >> latencies you have been observing in the test) > >> 3. Block cache consists of 3 parts (25% young generation, 50% - > tenured, 25% - permanent). Permanent part is for CF with > >> IN_MEMORY = true (you can specify this when you create CF). Block > first stored in 'young gen' space, then gets promoted to 'tenured gen' space > >> (or gets evicted). May be your 'perm gen' space is underutilized? This > is exact 25% of 4GB (1GB). Although HBase LruBlockCache should use all the > space allocated for block cache - > >> there is no guarantee (as usual). If you don have in_memory column > families you may decrease > >> > >> > >> > >> Best regards, > >> Vladimir Rodionov > >> Principal Platform Engineer > >> Carrier IQ, www.carrieriq.com > >> e-mail: vrodio...@carrieriq.com > >> > >> ________________________________________ > >> From: Saurabh Yahoo [saurabh...@yahoo.com] > >> Sent: Wednesday, August 28, 2013 5:10 AM > >> To: user@hbase.apache.org > >> Subject: experiencing high latency for few reads in HBase > >> > >> Hi, > >> > >> We are running a stress test in our 5 node cluster and we are getting > the expected mean latency of 10ms. But we are seeing around 20 reads out of > 25 million reads having latency more than 4 seconds. Can anyone provide the > insight what we can do to meet below second SLA for each and every read? > >> > >> We observe the following things - > >> > >> 1. Reads are evenly distributed among 5 nodes. CPUs remain under 5% > utilized. > >> > >> 2. We have 4gb block cache (30% block cache out of 12gb) setup. 3gb > block cache got filled up but around 1gb remained free. There are a large > number of cache eviction. > >> > >> Questions to experts - > >> > >> 1. If there are still 1gb of free block cache available, why is hbase > evicting the block from cache? > >> > >> 4. We are seeing memory went up to 10gb three times before dropping > sharply to 5gb. > >> > >> Any help is highly appreciable, > >> > >> Thanks, > >> Saurabh. > >> > >> Confidentiality Notice: The information contained in this message, > including any attachments hereto, may be confidential and is intended to be > read only by the individual or entity to whom this message is addressed. If > the reader of this message is not the intended recipient or an agent or > designee of the intended recipient, please note that any review, use, > disclosure or distribution of this message or its attachments, in any form, > is strictly prohibited. If you have received this message in error, please > immediately notify the sender and/or notificati...@carrieriq.com and > delete or destroy any copy of this message and its attachments. > -- Adrien Mogenet http://www.borntosegfault.com
