Thanks for the suggestions! The total size of the bucket cache is 72.00 GB. We generally have close to half of that used when the issue happens. We are using only one file path for the bucket cache. We will try using multiple paths and also adding an additional disk to our region-servers as suggested.
When looking through the HBase Jira I came across this ticket -- https://issues.apache.org/jira/browse/HBASE-16630 that affects the version of HBase that we are running. From what I can tell, this bug + fix looks like it only applies when the Bucket Cache is running in memory. Is there an equivalent bug + fix for a Bucket Cache running in file mode? Thanks, Srinidhi On 4/5/19, 5:14 AM, "Anoop John" <anoop.hb...@gmail.com> wrote: Hi Srinidhi You have File mode bucket cache. What is the size of the cache? You configure single file path for cache or 1+ paths? If former, splitting the cache into multiple files (paths can be given as comma separated in the config) will help? Anoop On Fri, Apr 5, 2019 at 2:58 AM Srinidhi Muppalla <srinid...@trulia.com> wrote: > After some more digging, I discovered that during the time that the RS is > stuck the kernel message buffer outputted only this message > > "[1031214.108110] XFS: java(6522) possible memory allocation deadlock size > 32944 in kmem_alloc (mode:0x2400240)" > > From my reading online, the cause of this error appears to generally be > excessive memory and file fragmentation. We haven't changed the mslab > config and we are running HBase 1.3.0 so it should be running by default. > The issue tends to arise consistently and regularly (every 10 or so days) > and once one node is affected other nodes start to follow after a few > hours. What could be causing this to happen and is there any way to prevent > or minimize fragmentation? > > Best, > Srinidhi > > On 3/29/19, 11:02 AM, "Srinidhi Muppalla" <srinid...@trulia.com> wrote: > > Stack and Ram, > > Attached the thread dumps. 'Jstack normal' is the normal node. 'Jstack > problematic' was taken when the node was stuck. > > We don't have full I/O stats for the problematic node. Unfortunately, > it was impacting production so we had to recreate the cluster as soon as > possible and couldn't get full data. I attached the dashboards with the > wait I/O and other CPU stats. Thanks for helping look into the issue! > > Best, > Srinidhi > > > > On 3/28/19, 2:41 PM, "Stack" <st...@duboce.net> wrote: > > Mind putting up a thread dump? > > How many spindles? > > If you compare the i/o stats between a good RS and a stuck one, > how do they > compare? > > Thanks, > S > > > On Wed, Mar 27, 2019 at 11:57 AM Srinidhi Muppalla < > srinid...@trulia.com> > wrote: > > > Hello, > > > > We've noticed an issue in our HBase cluster where one of the > > region-servers has a spike in I/O wait associated with a spike > in Load for > > that node. As a result, our request times to the cluster increase > > dramatically. Initially, we suspected that we were experiencing > > hotspotting, but even after temporarily blocking requests to the > highest > > volume regions on that region-servers the issue persisted. > Moreover, when > > looking at request counts to the regions on the region-server > from the > > HBase UI, they were not particularly high and our own > application level > > metrics on the requests we were making were not very high > either. From > > looking at a thread dump of the region-server, it appears that > our get and > > scan requests are getting stuck when trying to read from the > blocks in our > > bucket cache leaving the threads in a 'runnable' state. For > context, we are > > running HBase 1.30 on a cluster backed by S3 running on EMR and > our bucket > > cache is running in File mode. Our region-servers all have SSDs. > We have a > > combined cache with the L1 standard LRU cache and the L2 file > mode bucket > > cache. Our Bucket Cache utilization is less than 50% of the > allocated space. > > > > We suspect that part of the issue is our disk space utilization > on the > > region-server as our max disk space utilization also increased > as this > > happened. What things can we do to minimize disk space > utilization? The > > actual HFiles are on S3 -- only the cache, application logs, and > write > > ahead logs are on the region-servers. Other than the disk space > > utilization, what factors could cause high I/O wait in HBase and > is there > > anything we can do to minimize it? > > > > Right now, the only thing that works is terminating and > recreating the > > cluster (which we can do safely because it's S3 backed). > > > > Thanks! > > Srinidhi > > > > > > >
