[jira] [Created] (HBASE-11433) LruBlockCache does not respect its configurable parameters
Shengzhe Yao created HBASE-11433: Summary: LruBlockCache does not respect its configurable parameters Key: HBASE-11433 URL: https://issues.apache.org/jira/browse/HBASE-11433 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.98.3, 0.96.2, 0.99.0 Reporter: Shengzhe Yao Assignee: Shengzhe Yao Fix For: 0.99.0 We've upgraded our production cluster from 0.94.15 to 0.96.2 few days ago and observed increased GC frequency and occasionally full GC (we never had full GC before with G1 GC), which leads to famous juliet pause... After digging into several HBase metrics, we've found that block cache used much higher memory in 0.96. It turns out due to patch: HBASE-6312, which not only make a few block cache parameter configurable, but also change their default values! It is obvious that we need to set these parameters back to the old value before considering reduce block cache size or tuning our GC. However, we are surprised that there is no change in regionserver side and we are still observing high block cache usage. At the end of the day, it seems in CacheConfig.java, we initialize LruBlockCache with default constructor: LruBlockCache(long maxSize, long blockSize), which underlying always use the default values. We think this is a bug and we should always use another constructor: LruBlockCache(long maxSize, long blockSize, boolean evictionThread, Configuration conf) in CacheConfig.java We made the change and tested on one of our servers, it works and now GC problem disappears. Of course, we have to review our hbase and GC configurations and find the best configuration under 0.96 for our application. But first, we feel the constructor misuse in CacheConfig.java should be fixed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11487) ScanResponse carries non-zero cellblock for CloseScanRequest (ScanRequest with close_scanner = true)
Shengzhe Yao created HBASE-11487: Summary: ScanResponse carries non-zero cellblock for CloseScanRequest (ScanRequest with close_scanner = true) Key: HBASE-11487 URL: https://issues.apache.org/jira/browse/HBASE-11487 Project: HBase Issue Type: Bug Components: IPC/RPC, regionserver Affects Versions: 0.96.2, 0.99.0, 2.0.0 Reporter: Shengzhe Yao Assignee: Shengzhe Yao Priority: Minor Fix For: 2.0.0 After upgrading hbase from 0.94 to 0.96, we've found that our asynchbase client keep throwing errors during normal scan. It turns out these errors are due to Scanner.close call in asynchbase. Since asynchbase assumes the ScanResponse of CloseScannerRequest should never carry any cellblocks, it will throw an exception if there is a violation. In the asynchbase client (1.5.0), it constructs a CloseScannerRequest in the following way, ScanRequest.newBuilder() .setScannerId(scanner_id) .setCloseScanner(true) .build(); Note, it does not set numOfRows, which kind of make sense. Why a close scanner request cares about number of rows to scan ? However, after narrowing down the CloseScannerRequest code path, it seems the issue is on regionserver side. In RsRpcServices.scan, we always init numOfRows to scan to 1 and we do this even for ScanRequest with close_scanner = true. This causes response for CloseScannerRequest will carry a cellBlock (if scan stops before the end row and this could happen in many normal scenarios) There are two fixes, either we always set numOfRows in asynchbase client side when constructing a CloseScannerRequest or we fix the default value in the server side. >From a hbase client side point of view, it seems make less sense that server >will send you a cellBlock for your close scanner request, unless the request >explicitly asks for. We've made the change in our server code and the asynchbase client errors goes away. In addition to this issue, I want to know if we have any specifications for our hbase rpc. Like if close_scanner = true in ScanRequest and numOfRows is not set, ScanResponse guarantees that there is no cellBlock in the response. Since we moved to protobuf and many fields are optional for compatibility consideration, it might be helpful to have such specification which helps people to develop code that depends on hbase rpc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11600) DataInputputStream and DoubleOutputStream are no longer being used
Shengzhe Yao created HBASE-11600: Summary: DataInputputStream and DoubleOutputStream are no longer being used Key: HBASE-11600 URL: https://issues.apache.org/jira/browse/HBASE-11600 Project: HBase Issue Type: Task Components: io Reporter: Shengzhe Yao Assignee: Shengzhe Yao Priority: Trivial Fix For: 2.0.0 hbase-server/src/main/java/org/apache/hadoop/hbase/io/DataInputInputStream.java and hbase-server/src/main/java/org/apache/hadoop/hbase/io/DoubleOutputStream.java seems no longer being used. -- This message was sent by Atlassian JIRA (v6.2#6252)
