[ https://issues.apache.org/jira/browse/HBASE-16747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15602938#comment-15602938 ]
Hadoop QA commented on HBASE-16747: ----------------------------------- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 11 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 6s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 40s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 22s {color} | {color:green} master passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 48s {color} | {color:red} hbase-server in master has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s {color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 1s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 39s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 22s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 29m 22s {color} | {color:green} Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 21s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 46s {color} | {color:red} hbase-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 4s {color} | {color:red} hbase-server generated 2 new + 0 unchanged - 1 fixed = 2 total (was 1) {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 30s {color} | {color:red} hbase-server generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1) {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 50s {color} | {color:green} hbase-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 7s {color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 62m 41s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hbase-common | | | Uninitialized read of cell in new org.apache.hadoop.hbase.CellUtil$TagRewriteCell(Cell, byte[]) At CellUtil.java:new org.apache.hadoop.hbase.CellUtil$TagRewriteCell(Cell, byte[]) At CellUtil.java:[line 395] | | FindBugs | module:hbase-server | | | org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion$BatchOperation) does not release lock on all paths At HRegion.java:on all paths At HRegion.java:[line 3273] | | | Integral division result cast to double or float in org.apache.hadoop.hbase.regionserver.HeapMemoryManager$HeapMemoryTunerChore.tune() At HeapMemoryManager.java:double or float in org.apache.hadoop.hbase.regionserver.HeapMemoryManager$HeapMemoryTunerChore.tune() At HeapMemoryManager.java:[line 303] | | Failed junit tests | hadoop.hbase.regionserver.TestHeapMemoryManager | | | hadoop.hbase.TestTagRewriteCell | | | hadoop.hbase.regionserver.TestSplitTransaction | | | hadoop.hbase.regionserver.TestRegionMergeTransaction | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12834989/HBASE-16747.patch | | JIRA Issue | HBASE-16747 | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux a9e30cc89fe9 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 988d1f9 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | findbugs | https://builds.apache.org/job/PreCommit-HBASE-Build/4166/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html | | findbugs | https://builds.apache.org/job/PreCommit-HBASE-Build/4166/artifact/patchprocess/new-findbugs-hbase-common.html | | findbugs | https://builds.apache.org/job/PreCommit-HBASE-Build/4166/artifact/patchprocess/new-findbugs-hbase-server.html | | javadoc | https://builds.apache.org/job/PreCommit-HBASE-Build/4166/artifact/patchprocess/diff-javadoc-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/PreCommit-HBASE-Build/4166/artifact/patchprocess/patch-unit-hbase-server.txt | | unit test logs | https://builds.apache.org/job/PreCommit-HBASE-Build/4166/artifact/patchprocess/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/4166/testReport/ | | modules | C: hbase-common hbase-server U: . | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/4166/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > Track memstore data size and heap overhead separately > ------------------------------------------------------ > > Key: HBASE-16747 > URL: https://issues.apache.org/jira/browse/HBASE-16747 > Project: HBase > Issue Type: Sub-task > Components: regionserver > Reporter: Anoop Sam John > Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16747.patch, HBASE-16747.patch, > HBASE-16747_WIP.patch > > > We track the memstore size in 3 places. > 1. Global at RS level in RegionServerAccounting. This tracks all memstore's > size and used to calculate whether forced flushes needed because of global > heap pressure > 2. At region level in HRegion. This is sum of sizes of all memstores within > this region. This is used to decide whether region reaches flush size (128 MB) > 3. Segment level. This tracks the in memory flush/compaction decisions. > All these use the Cell's heap size which include the data bytes# as well as > Cell object heap overhead. Also we include the overhead because of addition > of Cells into Segment's data structures (Like CSLM). > Once we have off heap memstore, we will keep the cell data bytes in off heap > area. So we can not track both data size and heap overhead as one entity. We > need to separate them and track. > Proposal here is to track both cell data size and heap overhead separately at > global accounting layer. As of now we have only on heap memstore. So the > global memstore boundary checks will consider both (adds up and check against > global max memstore size) > Track cell data size alone (This can be on heap or off heap) in region level. > Region flushes use cell data size alone for the region flush decision. A > user configuring 128 MB as flush size, normally he will expect to get a 128MB > data flush size. But as we were including the heap overhead also, once the > flush happens, the actual data size getting flushed is way behind this 128 > MB. Now with this change we will behave more like what a user thinks. > Segment level in memory flush/compaction also considers cell data size alone. > But we will need to track the heap overhead also. (Once the in memory flush > or normal flush happens, we will have to adjust both cell data size and heap > overhead) -- This message was sent by Atlassian JIRA (v6.3.4#6332)