Where exactly in StoreScanner are you able to pinpoint the slowness to? You might try going deeper into StoreFileScanner -> HFileReaderImpl -> HFileBlock. In my experience, the code path follows there as you get closer to the FS, but the complexity of the code increases as well. The ultimate method just before HDFS is typically around FSReaderImpl.readAtOffset, in HFileBlock.java.
I'm unsure if it's available in your version, but histograms exist for fs access. Search JMX for FsReadTime. You could try seeing how many blocks are required to read the given rows. For large rows or rows that span multiple storefiles, this can affect performance a lot. An example of how to do that is here: https://github.com/apache/hbase/pull/4967 Also, are you sure you couldn't use async-profiler? We use this all the time in our very latency-sensitive production. It has no noticeable overhead in my experience and doesn't need any special dependencies. On Tue, Dec 5, 2023 at 3:46 PM Lars Francke <lars.fran...@gmail.com> wrote: > Hi, > > I am debugging an issue where we see some Get requests taking 2-5s. > We do see "responseTooSlow" etc. and this is in an environment where I > cannot run a Profiler but I _can_ run modified code. > > So what I did was I added a stupid "MethodTimer"[1] which records how > long certain operations take at various points in the code (e.g. [2]). > I've been doing this a few rounds and have now arrived at the StoreScanner. > > I'm wondering if anyone has better ideas on how to diagnose this? > I am a HBase committer but I haven't been able to keep up with the > changes in the last 5-6 years so I'm not too familiar with the inner > workings anymore and would appreciate a hint. > > I suspect it is slowness related to storage access. > I was not able to find any logs or tweaks to log "slow storage" > access, does such a thing exist? > And something else that'd help me: Can anyone point me (if it exists) > at the (vicinity of the) code that actually reads from HDFS at the > end? There are so many layers. > > Thank you! > > Cheers, > Lars > > > [1] < > https://github.com/stackabletech/docker-images/blob/8349f29f8aded8a01a8d1dbf7a90776ede1764ca/hbase/stackable/patches/2.4.12/005-STACKABLE-profiling-2.4.12.patch#L150C5-L150C5 > > > [2] < > https://github.com/stackabletech/docker-images/blob/8349f29f8aded8a01a8d1dbf7a90776ede1764ca/hbase/stackable/patches/2.4.12/005-STACKABLE-profiling-2.4.12.patch#L289-L297 > > >
