Please schedule compaction on SYSTEM.STATS table to clear the old entries. On Thu, Sep 19, 2019 at 1:48 PM Stepan Migunov < stepan.migu...@firstlinesoftware.com> wrote:
> Thanks, Josh. The problem was really related to reading the SYSTEM.STATS > table. > There were only 8,000 rows in the table, but COUNT took more than 10 > minutes. I noticed that the storage files (34) had a total size of 10 GB. > > DELETE FROM SYSTEM.STATS did not help - the storage files are still 10 GB, > and COUNT took a long time. > Then I truncated the table from the hbase shell. And this fixed the > problem - after UPDATE STATS for each table, everything works fine. > > Are there any known issues with SYSTEM.STATS table? Apache Phoenix 4.13.1 > with 15 Region Servers. > > -----Original Message----- > From: Josh Elser [mailto:els...@apache.org] > Sent: Tuesday, September 17, 2019 5:16 PM > To: user@phoenix.apache.org > Subject: Re: Performance degradation on query analysis > > Can you share the output you see from the EXPLAIN? Does it differ between > times it's "fast" and times it's "slow"? > > Sharing the table(s) DDL statements would also help, along with the shape > and version of your cluster (e.g. Apache Phoenix 4.14.2 with 8 > RegionServers). > > Spit-balling ideas: > > Could be reads over the SYSTEM.CATALOG table or the SYSTEM.STATS table. > > Have you looked more coarsely at the RegionServer logs/metrics? Any obvious > saturation issues (e.g. handlers consumed, JVM GC pauses, host CPU > saturation)? > > Turn on DEBUG log4j client side (beware of chatty ZK logging) and see if > there's something obvious from when the EXPLAIN is slow. > > On 9/17/19 3:58 AM, Stepan Migunov wrote: > > Hi > > We have an issue with our production environment - from time to time we > > notice a significant performance degradation for some queries. The > strange > > thing is that the EXPLAIN operator for these queries takes the same time > > as queries execution (5 minutes or more). So, I guess, the issue is > > related to query's analysis but not data extraction. Is it possible that > > issue is related to SYSTEM.STATS access problem? Any other ideas? > > >
