[jira] [Commented] (HBASE-19844) Shell should support to flush by regionserver
[ https://issues.apache.org/jira/browse/HBASE-19844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359881#comment-16359881 ] Hadoop QA commented on HBASE-19844: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 37s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 44s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m 23s{color} | {color:red} The patch generated 7 new + 383 unchanged - 2 fixed = 390 total (was 385) {color} | | {color:red}-1{color} | {color:red} ruby-lint {color} | {color:red} 0m 3s{color} | {color:red} The patch generated 4 new + 729 unchanged - 3 fixed = 733 total (was 732) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 21s{color} | {color:green} hbase-shell in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 10s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 18m 2s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 | | JIRA Issue | HBASE-19844 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12910084/HBASE-19844.master.001.patch | | Optional Tests | asflicense javac javadoc unit rubocop ruby_lint | | uname | Linux e6ab072452de 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 15:49:21 UTC 2017 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 0593dda663 | | maven | version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) | | Default Java | 1.8.0_151 | | rubocop | v0.52.1 | | rubocop | https://builds.apache.org/job/PreCommit-HBASE-Build/11483/artifact/patchprocess/diff-patch-rubocop.txt | | ruby-lint | v2.3.1 | | ruby-lint | https://builds.apache.org/job/PreCommit-HBASE-Build/11483/artifact/patchprocess/diff-patch-ruby-lint.txt | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/11483/testReport/ | | Max. process+thread count | 1864 (vs. ulimit of 1) | | modules | C: hbase-shell U: hbase-shell | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/11483/console | | Powered by | Apache Yetus 0.7.0 http://yetus.apache.org | This message was automatically generated. > Shell should support to flush by regionserver > - > > Key: HBASE-19844 > URL: https://issues.apache.org/jira/browse/HBASE-19844 > Project: HBase > Issue Type: New Feature > Components: shell >Reporter: Chia-Ping Tsai >Assignee: Reid Chan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-19844.master.001.patch > > > HBASE-4224 add a method to admin that can do the flush by regionserver. As > with other Admin methods, we should enable shell to use the flush method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-18294: -- Attachment: HBASE-18294.01.patch > Reduce global heap pressure: flush based on heap occupancy > -- > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.02.patch, HBASE-18294.03.patch, > HBASE-18294.04.patch, HBASE-18294.05.patch, HBASE-18294.06.patch, > HBASE-18294.07.patch, HBASE-18294.07.patch, HBASE-18294.08.patch, > HBASE-18294.09.patch, HBASE-18294.10.patch, HBASE-18294.11.patch, > HBASE-18294.11.patch, HBASE-18294.12.patch, HBASE-18294.13.patch, > HBASE-18294.15.patch, HBASE-18294.16.patch, HBASE-18294.master.01.patch, > HBASE-18294.master.01.patch > > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19082) Reject read/write from client but accept write from replication in state S
[ https://issues.apache.org/jira/browse/HBASE-19082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19082: -- Labels: (was: HBASE-19064) > Reject read/write from client but accept write from replication in state S > -- > > Key: HBASE-19082 > URL: https://issues.apache.org/jira/browse/HBASE-19082 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-19082-HBASE-19064.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19082) Reject read/write from client but accept write from replication in state S
[ https://issues.apache.org/jira/browse/HBASE-19082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19082: -- Attachment: HBASE-19082-HBASE-19064.patch > Reject read/write from client but accept write from replication in state S > -- > > Key: HBASE-19082 > URL: https://issues.apache.org/jira/browse/HBASE-19082 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Duo Zhang >Priority: Major > Labels: HBASE-19064 > Attachments: HBASE-19082-HBASE-19064.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19082) Reject read/write from client but accept write from replication in state S
[ https://issues.apache.org/jira/browse/HBASE-19082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19082: -- Status: Patch Available (was: Open) > Reject read/write from client but accept write from replication in state S > -- > > Key: HBASE-19082 > URL: https://issues.apache.org/jira/browse/HBASE-19082 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-19082-HBASE-19064.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-19082) Reject read/write from client but accept write from replication in state S
[ https://issues.apache.org/jira/browse/HBASE-19082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HBASE-19082: - Assignee: Duo Zhang Fix Version/s: (was: 3.0.0) > Reject read/write from client but accept write from replication in state S > -- > > Key: HBASE-19082 > URL: https://issues.apache.org/jira/browse/HBASE-19082 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Labels: HBASE-19064 > Attachments: HBASE-19082-HBASE-19064.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19959) How much RAM space is to be really consumed by the memstore?
[ https://issues.apache.org/jira/browse/HBASE-19959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359868#comment-16359868 ] Chance Li commented on HBASE-19959: --- To flush all the data to make the memstore size is 0 . > How much RAM space is to be really consumed by the memstore? > > > Key: HBASE-19959 > URL: https://issues.apache.org/jira/browse/HBASE-19959 > Project: HBase > Issue Type: Brainstorming > Components: regionserver >Reporter: Chance Li >Priority: Minor > > Let's consider this scenario where memstoreLAB and ChunkPool is enable and > max memstore size is 10G, and after some time all pooled chunk have been > created, then flush all data, now memstore size is 0 but RAM actually have > consumed 10G, then continue writing big cell which will not use the chunk > pool but jvm heap, then memstore size will be increased to 10G(maybe more > because overhead). now we can see RAM actually consumed 20G (10G of pooled > chunk + 10G java objects), but the max memstore size is only 10G. > what I say is the max memstore size not only take care about the cell "size" > but also RAM really used. This will be a strict memory management: the max > memstore size limit the RAM space which the memstore or related module can be > used. > This really rarely occured. It's just for a robust memory managemant > semantically. > What do you think? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-18294: -- Attachment: HBASE-18294.01.patch > Reduce global heap pressure: flush based on heap occupancy > -- > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.02.patch, HBASE-18294.03.patch, HBASE-18294.04.patch, > HBASE-18294.05.patch, HBASE-18294.06.patch, HBASE-18294.07.patch, > HBASE-18294.07.patch, HBASE-18294.08.patch, HBASE-18294.09.patch, > HBASE-18294.10.patch, HBASE-18294.11.patch, HBASE-18294.11.patch, > HBASE-18294.12.patch, HBASE-18294.13.patch, HBASE-18294.15.patch, > HBASE-18294.16.patch, HBASE-18294.master.01.patch, HBASE-18294.master.01.patch > > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19972) Should rethrow the RetriesExhaustedWithDetailsException when failed to apply the batch in ReplicationSink
[ https://issues.apache.org/jira/browse/HBASE-19972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-19972: - Status: Patch Available (was: Open) > Should rethrow the RetriesExhaustedWithDetailsException when failed to apply > the batch in ReplicationSink > -- > > Key: HBASE-19972 > URL: https://issues.apache.org/jira/browse/HBASE-19972 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Critical > Fix For: 1.5.0, 2.0.0-beta-2, 1.4.2 > > Attachments: HBASE-19972-branch-1.4.patch, HBASE-19972.v1.patch > > > As [~Apache9] said in HBASE-12091. > In ReplicationSink#batch,we swallow the RetriesExhaustedWithDetailsException > except > TableNotFoundException, actually, should rethrow the exception. > {code:java} > try { > Connection connection = getConnection(); > table = connection.getTable(tableName); > for (List rows : allRows) { > table.batch(rows); > } > } catch (RetriesExhaustedWithDetailsException rewde) { > for (Throwable ex : rewde.getCauses()) { > if (ex instanceof TableNotFoundException) { > throw new TableNotFoundException("'"+tableName+"'"); > } > } > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19844) Shell should support to flush by regionserver
[ https://issues.apache.org/jira/browse/HBASE-19844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359906#comment-16359906 ] Ted Yu commented on HBASE-19844: Can you look at "flush should work" in hbase-shell/src/test//ruby/hbase/admin_test.rb and add a similar test ? > Shell should support to flush by regionserver > - > > Key: HBASE-19844 > URL: https://issues.apache.org/jira/browse/HBASE-19844 > Project: HBase > Issue Type: New Feature > Components: shell >Reporter: Chia-Ping Tsai >Assignee: Reid Chan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-19844.master.001.patch, > HBASE-19844.master.002.patch > > > HBASE-4224 add a method to admin that can do the flush by regionserver. As > with other Admin methods, we should enable shell to use the flush method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19972) Should rethrow the RetriesExhaustedWithDetailsException when failed to apply the batch in ReplicationSink
[ https://issues.apache.org/jira/browse/HBASE-19972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359910#comment-16359910 ] Hadoop QA commented on HBASE-19972: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} branch-1.4 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 47s{color} | {color:green} branch-1.4 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s{color} | {color:green} branch-1.4 passed with JDK v1.8.0_162 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s{color} | {color:green} branch-1.4 passed with JDK v1.7.0_171 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 19s{color} | {color:green} branch-1.4 passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m 56s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 12s{color} | {color:green} branch-1.4 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s{color} | {color:green} branch-1.4 passed with JDK v1.8.0_162 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s{color} | {color:green} branch-1.4 passed with JDK v1.7.0_171 {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s{color} | {color:green} the patch passed with JDK v1.8.0_162 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s{color} | {color:green} the patch passed with JDK v1.7.0_171 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m 36s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 8m 48s{color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1 2.5.2 2.6.5 2.7.4. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s{color} | {color:green} the patch passed with JDK v1.8.0_162 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s{color} | {color:green} the patch passed with JDK v1.7.0_171 {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 98m 23s{color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}125m 50s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hbase.util.TestHBaseFsck | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:74e3133 | | JIRA Issue | HBASE-19972 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12910075/HBASE-19972-branch-1.4.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars
[jira] [Updated] (HBASE-19844) Shell should support to flush by regionserver
[ https://issues.apache.org/jira/browse/HBASE-19844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-19844: -- Attachment: HBASE-19844.master.001.patch > Shell should support to flush by regionserver > - > > Key: HBASE-19844 > URL: https://issues.apache.org/jira/browse/HBASE-19844 > Project: HBase > Issue Type: New Feature > Components: shell >Reporter: Chia-Ping Tsai >Assignee: Reid Chan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-19844.master.001.patch > > > HBASE-4224 add a method to admin that can do the flush by regionserver. As > with other Admin methods, we should enable shell to use the flush method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-18294: -- Attachment: HBASE-18294.master.01.patch > Reduce global heap pressure: flush based on heap occupancy > -- > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.02.patch, > HBASE-18294.03.patch, HBASE-18294.04.patch, HBASE-18294.05.patch, > HBASE-18294.06.patch, HBASE-18294.07.patch, HBASE-18294.07.patch, > HBASE-18294.08.patch, HBASE-18294.09.patch, HBASE-18294.10.patch, > HBASE-18294.11.patch, HBASE-18294.11.patch, HBASE-18294.12.patch, > HBASE-18294.13.patch, HBASE-18294.15.patch, HBASE-18294.16.patch, > HBASE-18294.master.01.patch, HBASE-18294.master.01.patch > > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19844) Shell should support to flush by regionserver
[ https://issues.apache.org/jira/browse/HBASE-19844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-19844: -- Status: Patch Available (was: Open) Already checked shell command on local machine. > Shell should support to flush by regionserver > - > > Key: HBASE-19844 > URL: https://issues.apache.org/jira/browse/HBASE-19844 > Project: HBase > Issue Type: New Feature > Components: shell >Reporter: Chia-Ping Tsai >Assignee: Reid Chan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-19844.master.001.patch > > > HBASE-4224 add a method to admin that can do the flush by regionserver. As > with other Admin methods, we should enable shell to use the flush method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-18294: -- Status: Patch Available (was: Open) > Reduce global heap pressure: flush based on heap occupancy > -- > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.02.patch, HBASE-18294.03.patch, HBASE-18294.04.patch, > HBASE-18294.05.patch, HBASE-18294.06.patch, HBASE-18294.07.patch, > HBASE-18294.07.patch, HBASE-18294.08.patch, HBASE-18294.09.patch, > HBASE-18294.10.patch, HBASE-18294.11.patch, HBASE-18294.11.patch, > HBASE-18294.12.patch, HBASE-18294.13.patch, HBASE-18294.15.patch, > HBASE-18294.16.patch, HBASE-18294.master.01.patch, HBASE-18294.master.01.patch > > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19844) Shell should support to flush by regionserver
[ https://issues.apache.org/jira/browse/HBASE-19844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359890#comment-16359890 ] Reid Chan commented on HBASE-19844: --- Try to fix {{rubocop}} and {{ruby-lint}} > Shell should support to flush by regionserver > - > > Key: HBASE-19844 > URL: https://issues.apache.org/jira/browse/HBASE-19844 > Project: HBase > Issue Type: New Feature > Components: shell >Reporter: Chia-Ping Tsai >Assignee: Reid Chan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-19844.master.001.patch, > HBASE-19844.master.002.patch > > > HBASE-4224 add a method to admin that can do the flush by regionserver. As > with other Admin methods, we should enable shell to use the flush method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19844) Shell should support to flush by regionserver
[ https://issues.apache.org/jira/browse/HBASE-19844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-19844: -- Attachment: HBASE-19844.master.002.patch > Shell should support to flush by regionserver > - > > Key: HBASE-19844 > URL: https://issues.apache.org/jira/browse/HBASE-19844 > Project: HBase > Issue Type: New Feature > Components: shell >Reporter: Chia-Ping Tsai >Assignee: Reid Chan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-19844.master.001.patch, > HBASE-19844.master.002.patch > > > HBASE-4224 add a method to admin that can do the flush by regionserver. As > with other Admin methods, we should enable shell to use the flush method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359907#comment-16359907 ] Hadoop QA commented on HBASE-18294: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 1s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 17 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 34s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 52s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 32s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 7m 15s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 37s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 17s{color} | {color:red} hbase-server: The patch generated 1 new + 806 unchanged - 7 fixed = 807 total (was 813) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 41s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 18m 44s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 16s{color} | {color:green} hbase-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 55s{color} | {color:green} hbase-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 18m 59s{color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m 57s{color} | {color:green} hbase-mapreduce in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 51s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 81m 45s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hbase.io.TestHeapSize | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 | | JIRA Issue | HBASE-18294 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12910086/HBASE-18294.01.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 9a0ed54552aa 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 15:49:21 UTC 2017 x86_64 GNU/Linux | | Build tool |
[jira] [Commented] (HBASE-19844) Shell should support to flush by regionserver
[ https://issues.apache.org/jira/browse/HBASE-19844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359908#comment-16359908 ] Reid Chan commented on HBASE-19844: --- Thanks for reminder Ted. > Shell should support to flush by regionserver > - > > Key: HBASE-19844 > URL: https://issues.apache.org/jira/browse/HBASE-19844 > Project: HBase > Issue Type: New Feature > Components: shell >Reporter: Chia-Ping Tsai >Assignee: Reid Chan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-19844.master.001.patch, > HBASE-19844.master.002.patch > > > HBASE-4224 add a method to admin that can do the flush by regionserver. As > with other Admin methods, we should enable shell to use the flush method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19959) How much RAM space is to be really consumed by the memstore?
[ https://issues.apache.org/jira/browse/HBASE-19959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359851#comment-16359851 ] Anastasia Braginsky commented on HBASE-19959: - OK, I may be understood it wrongly what you write. But then: {quote}Let's consider this scenario where memstoreLAB and ChunkPool is enable and max memstore size is 10G, and after some time all pooled chunk have been created, then flush all data, now memstore size is 0 {quote} Why do you say memstore size is 0? If you consume pool chunks and write there data, memstore must have value... > How much RAM space is to be really consumed by the memstore? > > > Key: HBASE-19959 > URL: https://issues.apache.org/jira/browse/HBASE-19959 > Project: HBase > Issue Type: Brainstorming > Components: regionserver >Reporter: Chance Li >Priority: Minor > > Let's consider this scenario where memstoreLAB and ChunkPool is enable and > max memstore size is 10G, and after some time all pooled chunk have been > created, then flush all data, now memstore size is 0 but RAM actually have > consumed 10G, then continue writing big cell which will not use the chunk > pool but jvm heap, then memstore size will be increased to 10G(maybe more > because overhead). now we can see RAM actually consumed 20G (10G of pooled > chunk + 10G java objects), but the max memstore size is only 10G. > what I say is the max memstore size not only take care about the cell "size" > but also RAM really used. This will be a strict memory management: the max > memstore size limit the RAM space which the memstore or related module can be > used. > This really rarely occured. It's just for a robust memory managemant > semantically. > What do you think? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19844) Shell should support to flush by regionserver
[ https://issues.apache.org/jira/browse/HBASE-19844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359900#comment-16359900 ] Hadoop QA commented on HBASE-19844: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 43s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 49s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m 26s{color} | {color:red} The patch generated 4 new + 383 unchanged - 2 fixed = 387 total (was 385) {color} | | {color:red}-1{color} | {color:red} ruby-lint {color} | {color:red} 0m 4s{color} | {color:red} The patch generated 3 new + 730 unchanged - 2 fixed = 733 total (was 732) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 28s{color} | {color:green} hbase-shell in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 10s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 18m 32s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 | | JIRA Issue | HBASE-19844 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12910088/HBASE-19844.master.002.patch | | Optional Tests | asflicense javac javadoc unit rubocop ruby_lint | | uname | Linux fe7b1cb17bb3 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh | | git revision | master / 0593dda663 | | maven | version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) | | Default Java | 1.8.0_151 | | rubocop | v0.52.1 | | rubocop | https://builds.apache.org/job/PreCommit-HBASE-Build/11484/artifact/patchprocess/diff-patch-rubocop.txt | | ruby-lint | v2.3.1 | | ruby-lint | https://builds.apache.org/job/PreCommit-HBASE-Build/11484/artifact/patchprocess/diff-patch-ruby-lint.txt | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/11484/testReport/ | | Max. process+thread count | 1846 (vs. ulimit of 1) | | modules | C: hbase-shell U: hbase-shell | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/11484/console | | Powered by | Apache Yetus 0.7.0 http://yetus.apache.org | This message was automatically generated. > Shell should support to flush by regionserver > - > > Key: HBASE-19844 > URL: https://issues.apache.org/jira/browse/HBASE-19844 > Project: HBase > Issue Type: New Feature > Components: shell >Reporter: Chia-Ping Tsai >Assignee: Reid Chan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-19844.master.001.patch, > HBASE-19844.master.002.patch > > > HBASE-4224 add a method to admin that can do the flush by regionserver. As > with other Admin methods, we should enable shell to use the flush method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19082) Reject read/write from client but accept write from replication in state S
[ https://issues.apache.org/jira/browse/HBASE-19082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359951#comment-16359951 ] Duo Zhang commented on HBASE-19082: --- Review board link: https://reviews.apache.org/r/65597/ > Reject read/write from client but accept write from replication in state S > -- > > Key: HBASE-19082 > URL: https://issues.apache.org/jira/browse/HBASE-19082 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-19082-HBASE-19064.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19082) Reject read/write from client but accept write from replication in state S
[ https://issues.apache.org/jira/browse/HBASE-19082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359980#comment-16359980 ] Hadoop QA commented on HBASE-19082: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} HBASE-19064 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 31s{color} | {color:green} HBASE-19064 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 42s{color} | {color:green} HBASE-19064 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 56s{color} | {color:green} HBASE-19064 passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m 35s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s{color} | {color:green} HBASE-19064 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 10s{color} | {color:green} The patch hbase-protocol-shaded passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 23s{color} | {color:green} The patch hbase-common passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 11s{color} | {color:green} The patch hbase-replication passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 12s{color} | {color:green} hbase-server: The patch generated 0 new + 478 unchanged - 5 fixed = 478 total (was 483) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 47s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 18m 27s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 30s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 18s{color} | {color:green} hbase-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s{color} | {color:green} hbase-replication in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red}103m 57s{color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense
[jira] [Commented] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359979#comment-16359979 ] Hadoop QA commented on HBASE-18294: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m 24s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 17 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 8s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 6s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 48s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 8m 7s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 32s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 34s{color} | {color:green} The patch hbase-common passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 35s{color} | {color:green} The patch hbase-client passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 25s{color} | {color:green} hbase-server: The patch generated 0 new + 806 unchanged - 7 fixed = 806 total (was 813) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 22s{color} | {color:green} The patch hbase-mapreduce passed checkstyle {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m 34s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 21m 36s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 39s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 29s{color} | {color:green} hbase-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 3s{color} | {color:green} hbase-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 11s{color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m 34s{color} | {color:green} hbase-mapreduce in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m 18s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}171m 58s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hbase.client.TestClientPushback |
[jira] [Commented] (HBASE-19950) Introduce a ColumnValueFilter
[ https://issues.apache.org/jira/browse/HBASE-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360030#comment-16360030 ] Mike Drob commented on HBASE-19950: --- nit: javadoc on createFilterFromArguments please > Introduce a ColumnValueFilter > - > > Key: HBASE-19950 > URL: https://issues.apache.org/jira/browse/HBASE-19950 > Project: HBase > Issue Type: Improvement > Components: Filters >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Minor > Attachments: HBASE-19950.master.001.patch, > HBASE-19950.master.002.patch, HBASE-19950.master.003.patch, > HBASE-19950.master.004.patch, HBASE-19950.master.005.patch, > HBASE-19950.master.006.patch, HBASE-19950.master.007.patch > > > Different from {{SingleColumnValueFilter}} which returns an entire row when > specified condition is matched, this new filter will return the matched cell > only. There're already some discussions in HBASE-19824. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-18294: -- Attachment: HBASE-18294.01.patch > Reduce global heap pressure: flush based on heap occupancy > -- > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.02.patch, > HBASE-18294.03.patch, HBASE-18294.04.patch, HBASE-18294.05.patch, > HBASE-18294.06.patch, HBASE-18294.07.patch, HBASE-18294.07.patch, > HBASE-18294.08.patch, HBASE-18294.09.patch, HBASE-18294.10.patch, > HBASE-18294.11.patch, HBASE-18294.11.patch, HBASE-18294.12.patch, > HBASE-18294.13.patch, HBASE-18294.15.patch, HBASE-18294.16.patch, > HBASE-18294.master.01.patch, HBASE-18294.master.01.patch > > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360087#comment-16360087 ] Eshcar Hillel commented on HBASE-18294: --- TestClientPushback is a flaky test, sometimes passes sometimes fails; I am not sure if its failure is related to this patch. In the test the load of the region is calculated by asking the region for its heap size, and there is a comment above the calculation saying {color:#808080}"get the current load on RS. Hopefully memstore isn't flushed since we wrote the the data"{color} {color:#33}(Hopefully ??!?), then this load is compared against the stats returned from the server and if they are not equal the test fails. The comment says "{color}{color:#808080}check to see we found some load on the memstore"{color}{color:#33} I guess it was meant to be "same load" but in some cases a flush causes the difference.{color}{color:#808080} {color} {color:#33}It could be that due to this patch the memstore is flushed earlier than before and so the test fails more frequently, but it seems to be unreasonable to *hope* for no flush during the test.{color} {color:#33}My solution was simply to double the flush size for this test. Any other more elegant suggestions for handling this?{color} > Reduce global heap pressure: flush based on heap occupancy > -- > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.01.patch, HBASE-18294.01.patch, > HBASE-18294.01.patch, HBASE-18294.02.patch, HBASE-18294.03.patch, > HBASE-18294.04.patch, HBASE-18294.05.patch, HBASE-18294.06.patch, > HBASE-18294.07.patch, HBASE-18294.07.patch, HBASE-18294.08.patch, > HBASE-18294.09.patch, HBASE-18294.10.patch, HBASE-18294.11.patch, > HBASE-18294.11.patch, HBASE-18294.12.patch, HBASE-18294.13.patch, > HBASE-18294.15.patch, HBASE-18294.16.patch, HBASE-18294.master.01.patch, > HBASE-18294.master.01.patch > > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19972) Should rethrow the RetriesExhaustedWithDetailsException when failed to apply the batch in ReplicationSink
[ https://issues.apache.org/jira/browse/HBASE-19972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360309#comment-16360309 ] Hadoop QA commented on HBASE-19972: --- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 3s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 12s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m 45s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m 42s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 19m 44s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green}100m 1s{color} | {color:green} hbase-server in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 21s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}141m 49s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 | | JIRA Issue | HBASE-19972 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12910131/HBASE-19972.v1.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux d49f11863bfc 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh | | git revision | master / 0593dda663 | | maven | version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) | | Default Java | 1.8.0_151 | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/11489/testReport/ | | Max. process+thread count | 4852 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/11489/console | | Powered by | Apache Yetus 0.7.0 http://yetus.apache.org | This message was automatically generated. > Should rethrow the RetriesExhaustedWithDetailsException when failed to apply > the batch in ReplicationSink >
[jira] [Updated] (HBASE-19950) Introduce a ColumnValueFilter
[ https://issues.apache.org/jira/browse/HBASE-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-19950: -- Attachment: HBASE-19950.master.008.patch > Introduce a ColumnValueFilter > - > > Key: HBASE-19950 > URL: https://issues.apache.org/jira/browse/HBASE-19950 > Project: HBase > Issue Type: Improvement > Components: Filters >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Minor > Attachments: HBASE-19950.master.001.patch, > HBASE-19950.master.002.patch, HBASE-19950.master.003.patch, > HBASE-19950.master.004.patch, HBASE-19950.master.005.patch, > HBASE-19950.master.006.patch, HBASE-19950.master.007.patch, > HBASE-19950.master.008.patch > > > Different from {{SingleColumnValueFilter}} which returns an entire row when > specified condition is matched, this new filter will return the matched cell > only. There're already some discussions in HBASE-19824. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19950) Introduce a ColumnValueFilter
[ https://issues.apache.org/jira/browse/HBASE-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360315#comment-16360315 ] Reid Chan commented on HBASE-19950: --- v8 patch adds javadoc for method #createFilterFromArguments(ArrayList) and #parseFrom(byte[]), and one more test case to address Anoop's concerns. Following snippet shows with {{#addColumn(...)}}, results will be limited to specified column, that's why i said {{#addColumn(...)}} is a strong constraint. While test case 7 without column constraint will scan more results. Both behaviors are in line with expectations, and i think this filter should read other CFs by default. Hoping that i make it clear, and address your concern, [~anoop.hbase]. {code} // 9. Test specified columns with FilterList(MUST_PASS_ONE) // (ColumnValueFilter, != f[1]:q5[1]:v[1]) || (QualifierFilter, = q5[0]) // Result is different from Test 7, because column is strongly constrained by specified columns Scan anotherScan = new Scan().addColumn(FAMILIES[1], QUALIFIERS_FIVE[1]) .setFilter(new FilterList(Operator.MUST_PASS_ONE, orFilters)); KeyValue[] expectedValues = { new KeyValue(ROWS_TWO[0], FAMILIES[1], QUALIFIERS_FIVE[1], VALUES[0]), new KeyValue(ROWS_TWO[1], FAMILIES[1], QUALIFIERS_FIVE[1], VALUES[0]), new KeyValue(ROWS_TWO[2], FAMILIES[1], QUALIFIERS_FIVE[1], VALUES[0]), new KeyValue(ROWS_TWO[3], FAMILIES[1], QUALIFIERS_FIVE[1], VALUES[0]) }; verifyScanFull(anotherScan, expectedValues); {code} BTW, please allow me to break the checkstyle of {{MethodLength}} exception from method {{#testColumnValueFilter()}}, becauses test cases should cover enough, and not let it block the commitment. :P > Introduce a ColumnValueFilter > - > > Key: HBASE-19950 > URL: https://issues.apache.org/jira/browse/HBASE-19950 > Project: HBase > Issue Type: Improvement > Components: Filters >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Minor > Attachments: HBASE-19950.master.001.patch, > HBASE-19950.master.002.patch, HBASE-19950.master.003.patch, > HBASE-19950.master.004.patch, HBASE-19950.master.005.patch, > HBASE-19950.master.006.patch, HBASE-19950.master.007.patch, > HBASE-19950.master.008.patch > > > Different from {{SingleColumnValueFilter}} which returns an entire row when > specified condition is matched, this new filter will return the matched cell > only. There're already some discussions in HBASE-19824. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19950) Introduce a ColumnValueFilter
[ https://issues.apache.org/jira/browse/HBASE-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-19950: -- Attachment: (was: HBASE-19950.master.008.patch) > Introduce a ColumnValueFilter > - > > Key: HBASE-19950 > URL: https://issues.apache.org/jira/browse/HBASE-19950 > Project: HBase > Issue Type: Improvement > Components: Filters >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Minor > Attachments: HBASE-19950.master.001.patch, > HBASE-19950.master.002.patch, HBASE-19950.master.003.patch, > HBASE-19950.master.004.patch, HBASE-19950.master.005.patch, > HBASE-19950.master.006.patch, HBASE-19950.master.007.patch > > > Different from {{SingleColumnValueFilter}} which returns an entire row when > specified condition is matched, this new filter will return the matched cell > only. There're already some discussions in HBASE-19824. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19950) Introduce a ColumnValueFilter
[ https://issues.apache.org/jira/browse/HBASE-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360316#comment-16360316 ] Reid Chan commented on HBASE-19950: --- I uploaded a wrong v8 patch... > Introduce a ColumnValueFilter > - > > Key: HBASE-19950 > URL: https://issues.apache.org/jira/browse/HBASE-19950 > Project: HBase > Issue Type: Improvement > Components: Filters >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Minor > Attachments: HBASE-19950.master.001.patch, > HBASE-19950.master.002.patch, HBASE-19950.master.003.patch, > HBASE-19950.master.004.patch, HBASE-19950.master.005.patch, > HBASE-19950.master.006.patch, HBASE-19950.master.007.patch, > HBASE-19950.master.008.patch > > > Different from {{SingleColumnValueFilter}} which returns an entire row when > specified condition is matched, this new filter will return the matched cell > only. There're already some discussions in HBASE-19824. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19950) Introduce a ColumnValueFilter
[ https://issues.apache.org/jira/browse/HBASE-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-19950: -- Attachment: HBASE-19950.master.008.patch > Introduce a ColumnValueFilter > - > > Key: HBASE-19950 > URL: https://issues.apache.org/jira/browse/HBASE-19950 > Project: HBase > Issue Type: Improvement > Components: Filters >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Minor > Attachments: HBASE-19950.master.001.patch, > HBASE-19950.master.002.patch, HBASE-19950.master.003.patch, > HBASE-19950.master.004.patch, HBASE-19950.master.005.patch, > HBASE-19950.master.006.patch, HBASE-19950.master.007.patch, > HBASE-19950.master.008.patch > > > Different from {{SingleColumnValueFilter}} which returns an entire row when > specified condition is matched, this new filter will return the matched cell > only. There're already some discussions in HBASE-19824. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19082) Reject read/write from client but accept write from replication in state S
[ https://issues.apache.org/jira/browse/HBASE-19082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19082: -- Attachment: HBASE-19082-HBASE-19064-v1.patch > Reject read/write from client but accept write from replication in state S > -- > > Key: HBASE-19082 > URL: https://issues.apache.org/jira/browse/HBASE-19082 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-19082-HBASE-19064-v1.patch, > HBASE-19082-HBASE-19064.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360164#comment-16360164 ] Hadoop QA commented on HBASE-18294: --- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 17 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 30s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 45s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 26s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 7m 20s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 38s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 28s{color} | {color:green} The patch hbase-common passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 30s{color} | {color:green} The patch hbase-client passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 24s{color} | {color:green} hbase-server: The patch generated 0 new + 806 unchanged - 7 fixed = 806 total (was 813) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 19s{color} | {color:green} The patch hbase-mapreduce passed checkstyle {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m 14s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 21m 44s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 37s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 39s{color} | {color:green} hbase-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 29s{color} | {color:green} hbase-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green}112m 19s{color} | {color:green} hbase-server in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m 29s{color} | {color:green} hbase-mapreduce in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m 19s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}181m 21s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce
[jira] [Commented] (HBASE-19082) Reject read/write from client but accept write from replication in state S
[ https://issues.apache.org/jira/browse/HBASE-19082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360181#comment-16360181 ] Duo Zhang commented on HBASE-19082: --- Let me check the failed UTs. > Reject read/write from client but accept write from replication in state S > -- > > Key: HBASE-19082 > URL: https://issues.apache.org/jira/browse/HBASE-19082 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-19082-HBASE-19064.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19938) Allow write request from replication but reject write request from user client when S state.
[ https://issues.apache.org/jira/browse/HBASE-19938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360351#comment-16360351 ] Zheng Hu commented on HBASE-19938: -- This issue is duplicate with HBASE-19082, and [~Apache9] is working on it, so close this issue now. > Allow write request from replication but reject write request from user > client when S state. > > > Key: HBASE-19938 > URL: https://issues.apache.org/jira/browse/HBASE-19938 > Project: HBase > Issue Type: Sub-task >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > > According the doc, we should reject write request when in S state ,however, > the replication data from master cluster will turn to a batch mutation > request (it's a write request). > So, for peer in S state, it should distinguish write request from > replication or from user .. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19972) Should rethrow the RetriesExhaustedWithDetailsException when failed to apply the batch in ReplicationSink
[ https://issues.apache.org/jira/browse/HBASE-19972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360357#comment-16360357 ] Duo Zhang commented on HBASE-19972: --- +1. > Should rethrow the RetriesExhaustedWithDetailsException when failed to apply > the batch in ReplicationSink > -- > > Key: HBASE-19972 > URL: https://issues.apache.org/jira/browse/HBASE-19972 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Critical > Fix For: 1.5.0, 2.0.0-beta-2, 1.4.2 > > Attachments: HBASE-19972-branch-1.4.patch, HBASE-19972.v1.patch, > HBASE-19972.v1.patch > > > As [~Apache9] said in HBASE-12091. > In ReplicationSink#batch,we swallow the RetriesExhaustedWithDetailsException > except > TableNotFoundException, actually, should rethrow the exception. > {code:java} > try { > Connection connection = getConnection(); > table = connection.getTable(tableName); > for (List rows : allRows) { > table.batch(rows); > } > } catch (RetriesExhaustedWithDetailsException rewde) { > for (Throwable ex : rewde.getCauses()) { > if (ex instanceof TableNotFoundException) { > throw new TableNotFoundException("'"+tableName+"'"); > } > } > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19972) Should rethrow the RetriesExhaustedWithDetailsException when failed to apply the batch in ReplicationSink
[ https://issues.apache.org/jira/browse/HBASE-19972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-19972: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Should rethrow the RetriesExhaustedWithDetailsException when failed to apply > the batch in ReplicationSink > -- > > Key: HBASE-19972 > URL: https://issues.apache.org/jira/browse/HBASE-19972 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Critical > Fix For: 1.5.0, 2.0.0-beta-2, 1.4.2 > > Attachments: HBASE-19972-branch-1.4.patch, HBASE-19972.v1.patch, > HBASE-19972.v1.patch > > > As [~Apache9] said in HBASE-12091. > In ReplicationSink#batch,we swallow the RetriesExhaustedWithDetailsException > except > TableNotFoundException, actually, should rethrow the exception. > {code:java} > try { > Connection connection = getConnection(); > table = connection.getTable(tableName); > for (List rows : allRows) { > table.batch(rows); > } > } catch (RetriesExhaustedWithDetailsException rewde) { > for (Throwable ex : rewde.getCauses()) { > if (ex instanceof TableNotFoundException) { > throw new TableNotFoundException("'"+tableName+"'"); > } > } > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19972) Should rethrow the RetriesExhaustedWithDetailsException when failed to apply the batch in ReplicationSink
[ https://issues.apache.org/jira/browse/HBASE-19972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360373#comment-16360373 ] Zheng Hu commented on HBASE-19972: -- Pushed to branch-1 & branch-1.4 & branch-2 & master , Thanks [~Apache9] for reviewing. > Should rethrow the RetriesExhaustedWithDetailsException when failed to apply > the batch in ReplicationSink > -- > > Key: HBASE-19972 > URL: https://issues.apache.org/jira/browse/HBASE-19972 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Critical > Fix For: 1.5.0, 2.0.0-beta-2, 1.4.2 > > Attachments: HBASE-19972-branch-1.4.patch, HBASE-19972.v1.patch, > HBASE-19972.v1.patch > > > As [~Apache9] said in HBASE-12091. > In ReplicationSink#batch,we swallow the RetriesExhaustedWithDetailsException > except > TableNotFoundException, actually, should rethrow the exception. > {code:java} > try { > Connection connection = getConnection(); > table = connection.getTable(tableName); > for (List rows : allRows) { > table.batch(rows); > } > } catch (RetriesExhaustedWithDetailsException rewde) { > for (Throwable ex : rewde.getCauses()) { > if (ex instanceof TableNotFoundException) { > throw new TableNotFoundException("'"+tableName+"'"); > } > } > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19554) AbstractTestDLS.testThreeRSAbort sometimes fails in pre commit
[ https://issues.apache.org/jira/browse/HBASE-19554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360401#comment-16360401 ] Duo Zhang commented on HBASE-19554: --- https://builds.apache.org/job/HBASE-Flaky-Tests/25832/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.master.TestDLSFSHLog-output.txt/*view*/ {noformat} 2018-02-12 04:56:20,895 DEBUG [PEWorker-16] procedure.ServerCrashProcedure(192): pid=132, state=RUNNABLE:SERVER_CRASH_PROCESS_META; ServerCrashProcedure server=asf911.gq1.ygridcore.net,41715,1518411358686, splitWal=true, meta=true; Processing hbase:meta that was on asf911.gq1.ygridcore.net,41715,1518411358686 2018-02-12 04:56:20,895 INFO [PEWorker-16] procedure2.ProcedureExecutor(1498): Initialized subprocedures=[{pid=135, ppid=132, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=asf911.gq1.ygridcore.net,41715,1518411358686, splitWal=true}] {noformat} Then there is no progress, so at last we time out. Let me add a thread dump when we are about to time out to see if we can find something. > AbstractTestDLS.testThreeRSAbort sometimes fails in pre commit > -- > > Key: HBASE-19554 > URL: https://issues.apache.org/jira/browse/HBASE-19554 > Project: HBase > Issue Type: Sub-task > Components: Recovery, wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19554.patch > > > https://builds.apache.org/job/PreCommit-HBASE-Build/10554/artifact/patchprocess/patch-unit-hbase-server.txt > The error message is a bit strange: > {quote} > [ERROR] testThreeRSAbort(org.apache.hadoop.hbase.master.TestDLSAsyncFSWAL) > Time elapsed: 20.627 s <<< ERROR! > org.apache.hadoop.hbase.TableNotFoundException: Region of > 'hbase:namespace,,1513320505933.451650152885a3b41d0b1110deca513c.' is > expected in the table of 'testThreeRSAbort', but hbase:meta says it is in the > table of 'hbase:namespace'. hbase:meta might be damaged. > {quote} > It fails for both FSHLog and AsyncFSWAL. Need to dig more. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19972) Should rethrow the RetriesExhaustedWithDetailsException when failed to apply the batch in ReplicationSink
[ https://issues.apache.org/jira/browse/HBASE-19972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360332#comment-16360332 ] Zheng Hu commented on HBASE-19972: -- All UT pass now, ping [~Apache9] & [~zghaobac] for review ... > Should rethrow the RetriesExhaustedWithDetailsException when failed to apply > the batch in ReplicationSink > -- > > Key: HBASE-19972 > URL: https://issues.apache.org/jira/browse/HBASE-19972 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Critical > Fix For: 1.5.0, 2.0.0-beta-2, 1.4.2 > > Attachments: HBASE-19972-branch-1.4.patch, HBASE-19972.v1.patch, > HBASE-19972.v1.patch > > > As [~Apache9] said in HBASE-12091. > In ReplicationSink#batch,we swallow the RetriesExhaustedWithDetailsException > except > TableNotFoundException, actually, should rethrow the exception. > {code:java} > try { > Connection connection = getConnection(); > table = connection.getTable(tableName); > for (List rows : allRows) { > table.batch(rows); > } > } catch (RetriesExhaustedWithDetailsException rewde) { > for (Throwable ex : rewde.getCauses()) { > if (ex instanceof TableNotFoundException) { > throw new TableNotFoundException("'"+tableName+"'"); > } > } > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-19938) Allow write request from replication but reject write request from user client when S state.
[ https://issues.apache.org/jira/browse/HBASE-19938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu resolved HBASE-19938. -- Resolution: Duplicate > Allow write request from replication but reject write request from user > client when S state. > > > Key: HBASE-19938 > URL: https://issues.apache.org/jira/browse/HBASE-19938 > Project: HBase > Issue Type: Sub-task >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > > According the doc, we should reject write request when in S state ,however, > the replication data from master cluster will turn to a batch mutation > request (it's a write request). > So, for peer in S state, it should distinguish write request from > replication or from user .. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19972) Should rethrow the RetriesExhaustedWithDetailsException when failed to apply the batch in ReplicationSink
[ https://issues.apache.org/jira/browse/HBASE-19972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360220#comment-16360220 ] Zheng Hu commented on HBASE-19972: -- Re-trigger Hadoop QA for master branch. The failed TestHBaseFsck ut in branch-1 is unrelated to this patch, If master branch Hadoop QA is OK, I'll commit this patch to master & branch-1 if no objection. > Should rethrow the RetriesExhaustedWithDetailsException when failed to apply > the batch in ReplicationSink > -- > > Key: HBASE-19972 > URL: https://issues.apache.org/jira/browse/HBASE-19972 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Critical > Fix For: 1.5.0, 2.0.0-beta-2, 1.4.2 > > Attachments: HBASE-19972-branch-1.4.patch, HBASE-19972.v1.patch, > HBASE-19972.v1.patch > > > As [~Apache9] said in HBASE-12091. > In ReplicationSink#batch,we swallow the RetriesExhaustedWithDetailsException > except > TableNotFoundException, actually, should rethrow the exception. > {code:java} > try { > Connection connection = getConnection(); > table = connection.getTable(tableName); > for (List rows : allRows) { > table.batch(rows); > } > } catch (RetriesExhaustedWithDetailsException rewde) { > for (Throwable ex : rewde.getCauses()) { > if (ex instanceof TableNotFoundException) { > throw new TableNotFoundException("'"+tableName+"'"); > } > } > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19326) Remove decommissioned servers from rsgroup
[ https://issues.apache.org/jira/browse/HBASE-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360247#comment-16360247 ] Xiaolin Ha commented on HBASE-19326: [~andrewcheng][~stack] In docs it said that 'The default group is not like other rsgroups in that it is dynamic. Its server list mirrors the current state of the cluster; i.e. if you shutdown a server that was part of the default rsgroup, and then do a get_rsgroup default to list its content in the shell, the server will no longer be listed. For non-default groups, though a mode may be offline, it will persist in the non-default group’s list of servers. But if you move the offline server from the non-default rsgroup to default, it will not show in the default list. It will just be dropped.' But when a regionserver is dead and master is restarted, the regionserver can be removed from the non-default rsgroup too. Is this conflict with the doc? Do you think dead regionservers can be removed either? > Remove decommissioned servers from rsgroup > -- > > Key: HBASE-19326 > URL: https://issues.apache.org/jira/browse/HBASE-19326 > Project: HBase > Issue Type: New Feature > Components: rsgroup >Affects Versions: 3.0.0, 2.0.0-beta-2 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Fix For: 1.4.0, 2.0.0-beta-1 > > Attachments: HBASE-19326-branch-1.patch, > HBASE-19326.branch-2.001.patch, HBASE-19326.master.001.patch, > HBASE-19326.master.002.patch, HBASE-19326.master.003.patch, > HBASE-19326.master.004.patch, HBASE-19326.master.005.patch > > > In HBASE-18131, we add an hbase shell command {{clear_deadservers}} to clear > deadserver list in ServerManager. > But rsgroup still contains these dead servers, so we should also remove dead > servers from the group information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19972) Should rethrow the RetriesExhaustedWithDetailsException when failed to apply the batch in ReplicationSink
[ https://issues.apache.org/jira/browse/HBASE-19972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-19972: - Attachment: HBASE-19972.v1.patch > Should rethrow the RetriesExhaustedWithDetailsException when failed to apply > the batch in ReplicationSink > -- > > Key: HBASE-19972 > URL: https://issues.apache.org/jira/browse/HBASE-19972 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Critical > Fix For: 1.5.0, 2.0.0-beta-2, 1.4.2 > > Attachments: HBASE-19972-branch-1.4.patch, HBASE-19972.v1.patch, > HBASE-19972.v1.patch > > > As [~Apache9] said in HBASE-12091. > In ReplicationSink#batch,we swallow the RetriesExhaustedWithDetailsException > except > TableNotFoundException, actually, should rethrow the exception. > {code:java} > try { > Connection connection = getConnection(); > table = connection.getTable(tableName); > for (List rows : allRows) { > table.batch(rows); > } > } catch (RetriesExhaustedWithDetailsException rewde) { > for (Throwable ex : rewde.getCauses()) { > if (ex instanceof TableNotFoundException) { > throw new TableNotFoundException("'"+tableName+"'"); > } > } > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19082) Reject read/write from client but accept write from replication in state S
[ https://issues.apache.org/jira/browse/HBASE-19082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360243#comment-16360243 ] Hadoop QA commented on HBASE-19082: --- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} HBASE-19064 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 22s{color} | {color:green} HBASE-19064 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 43s{color} | {color:green} HBASE-19064 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 55s{color} | {color:green} HBASE-19064 passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m 38s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s{color} | {color:green} HBASE-19064 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 9s{color} | {color:green} The patch hbase-protocol-shaded passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 23s{color} | {color:green} The patch hbase-common passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 11s{color} | {color:green} The patch hbase-replication passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 12s{color} | {color:green} hbase-server: The patch generated 0 new + 478 unchanged - 5 fixed = 478 total (was 483) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 47s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 18m 25s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 30s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 16s{color} | {color:green} hbase-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s{color} | {color:green} hbase-replication in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green}102m 52s{color} | {color:green} hbase-server in the patch passed. {color} | | {color:green}+1{color} | {color:green}