On Tue, Dec 5, 2023 at 10:15 PM Wellington Chevreuil <wellington.chevre...@gmail.com> wrote: > > > > > I suspect it is slowness related to storage access. > > I was not able to find any logs or tweaks to log "slow storage" > > access, does such a thing exist? > > > I believe I have seen warnings for slow syncs from the hdfs layer, but > don't remember any for the read path.
Likewise, yeah. I'll see if I can add something going forward, that might be useful. If I can find the right spots for it. > And something else that'd help me: Can anyone point me (if it exists) > > at the (vicinity of the) code that actually reads from HDFS at the > > end? There are so many layers. > > I think you might be interested in the HFileBlock.readBlockData(). I can > see we are open a FSDataInputStream there, then pass to some other methods, > using byte buffers and hadoop IOUtils to read from hdfs. Excellent, thank you! I'm sometimes wondering if it would make sense to document the architecture somewhere but the audience is small and it would need to be kept in sync. That's kept me from doing that so far. > > Em ter., 5 de dez. de 2023 às 20:46, Lars Francke <lars.fran...@gmail.com> > escreveu: > > > Hi, > > > > I am debugging an issue where we see some Get requests taking 2-5s. > > We do see "responseTooSlow" etc. and this is in an environment where I > > cannot run a Profiler but I _can_ run modified code. > > > > So what I did was I added a stupid "MethodTimer"[1] which records how > > long certain operations take at various points in the code (e.g. [2]). > > I've been doing this a few rounds and have now arrived at the StoreScanner. > > > > I'm wondering if anyone has better ideas on how to diagnose this? > > I am a HBase committer but I haven't been able to keep up with the > > changes in the last 5-6 years so I'm not too familiar with the inner > > workings anymore and would appreciate a hint. > > > > I suspect it is slowness related to storage access. > > I was not able to find any logs or tweaks to log "slow storage" > > access, does such a thing exist? > > And something else that'd help me: Can anyone point me (if it exists) > > at the (vicinity of the) code that actually reads from HDFS at the > > end? There are so many layers. > > > > Thank you! > > > > Cheers, > > Lars > > > > > > [1] < > > https://github.com/stackabletech/docker-images/blob/8349f29f8aded8a01a8d1dbf7a90776ede1764ca/hbase/stackable/patches/2.4.12/005-STACKABLE-profiling-2.4.12.patch#L150C5-L150C5 > > > > > [2] < > > https://github.com/stackabletech/docker-images/blob/8349f29f8aded8a01a8d1dbf7a90776ede1764ca/hbase/stackable/patches/2.4.12/005-STACKABLE-profiling-2.4.12.patch#L289-L297 > > > > >
