Thanks, Josh. The problem was really related to reading the SYSTEM.STATS table. There were only 8,000 rows in the table, but COUNT took more than 10 minutes. I noticed that the storage files (34) had a total size of 10 GB.
DELETE FROM SYSTEM.STATS did not help - the storage files are still 10 GB, and COUNT took a long time. Then I truncated the table from the hbase shell. And this fixed the problem - after UPDATE STATS for each table, everything works fine. Are there any known issues with SYSTEM.STATS table? Apache Phoenix 4.13.1 with 15 Region Servers. -----Original Message----- From: Josh Elser [mailto:els...@apache.org] Sent: Tuesday, September 17, 2019 5:16 PM To: user@phoenix.apache.org Subject: Re: Performance degradation on query analysis Can you share the output you see from the EXPLAIN? Does it differ between times it's "fast" and times it's "slow"? Sharing the table(s) DDL statements would also help, along with the shape and version of your cluster (e.g. Apache Phoenix 4.14.2 with 8 RegionServers). Spit-balling ideas: Could be reads over the SYSTEM.CATALOG table or the SYSTEM.STATS table. Have you looked more coarsely at the RegionServer logs/metrics? Any obvious saturation issues (e.g. handlers consumed, JVM GC pauses, host CPU saturation)? Turn on DEBUG log4j client side (beware of chatty ZK logging) and see if there's something obvious from when the EXPLAIN is slow. On 9/17/19 3:58 AM, Stepan Migunov wrote: > Hi > We have an issue with our production environment - from time to time we > notice a significant performance degradation for some queries. The strange > thing is that the EXPLAIN operator for these queries takes the same time > as queries execution (5 minutes or more). So, I guess, the issue is > related to query's analysis but not data extraction. Is it possible that > issue is related to SYSTEM.STATS access problem? Any other ideas? >
