They don't enough detail when you enable metrics on rpc? It should be easy enough hacking this in. What would you like to see on the query Jack? There'd be whether a Get/Delete/Put, etc., then what? We'd have to give this logging a different name than that of the general regionserver class because you'd never want to run a production system with it on; you'd only want to enable it say via the UI for a short period of time, and then turn it off, because it'd be a drag on perf and it would make for reams of output (Consider server doing 1k hits a second, or in your case Jack, 100k hits a second -- smile -- then we'd fill the disks in short order)
St.Ack On Sat, Apr 9, 2011 at 7:09 PM, Jack Levin <[email protected]> wrote: > Well, counts would help a lot, but actual queries displayed even more > so... For example I just found a bug in our code that runs in infinite > loop across only specific pages that cause region server CPU to > overrun. It would be easier for us to ID the bugs if we had all > queries logged. > > -Jack > > On Sat, Apr 9, 2011 at 6:02 PM, Ted Yu <[email protected]> wrote: >> I think HBASE-3647 was a good start but Jack asked might be a little more. >> BTW 3647 isn't in 0.90.2 >> >> Currently regionserver.jsp displays RegionServerMetrics for every region in >> an unsorted order w.r.t. read/write request counts. For any region server in >> production, there would be hundreds of regions, it is not realistic to spot >> hot regions in the current display. >> Do we need to add capability to sort the regions on regionserver.jsp >> according to read/write request counts ? >> >> Cheers >> >> On Sat, Apr 9, 2011 at 1:59 PM, Stack <[email protected]> wrote: >> >>> You mean per-region Jack? >>> >>> These went into TRUNK recently: >>> >>> HBASE-3647 Distinguish read and write request count in region >>> (Ted Yu via Stack) >>> HBASE-3704 Show per region request count in table.jsp >>> (Ted Yu via Stack) >>> >>> Would they help? >>> >>> St.Ack >>> >>> On Sat, Apr 9, 2011 at 12:22 PM, Jack Levin <[email protected]> wrote: >>> > It would be super nice to have RPC query call logging per region >>> > server. Sometimes we have 'hot' regions getting hit by users, and >>> > this drives the CPU on a particular RS server up, except we have no >>> > idea what table it might be. I could rig the tcpdump on port 60020, >>> > and log things with AWK, but that would be super hacky. An insight >>> > what traffic drives the cpu will allow us (or anyone else) model the >>> > tables to user patterns accordingly. >>> > >>> > thanks. >>> > >>> > -Jack >>> > >>> >> >
