[jira] [Commented] (HBASE-19964) TestWriteHeavyIncrementObserver fails
[ https://issues.apache.org/jira/browse/HBASE-19964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358125#comment-16358125 ] Duo Zhang commented on HBASE-19964: --- OK I think this is something happened before. Output the memstore size after the compaction check, it is not zero. This means that we fail to flush the table. Change to use HBaseTestingUtility.flush can fix the problem. But anyway, I'm still a little worried about the Admin.flush, we may fail to flush without any notification to users. > TestWriteHeavyIncrementObserver fails > - > > Key: HBASE-19964 > URL: https://issues.apache.org/jira/browse/HBASE-19964 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Reporter: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > > https://builds.apache.org/job/HBASE-Flaky-Tests/25704/artifact/hbase-examples/target/surefire-reports/org.apache.hadoop.hbase.coprocessor.example.TestWriteHeavyIncrementObserver-output.txt/*view*/ > I could reproduce it locally. Let me dig more. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19923) Reset peer state and config when refresh replication source failed
[ https://issues.apache.org/jira/browse/HBASE-19923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358144#comment-16358144 ] Guanghao Zhang commented on HBASE-19923: [~Apache9] Any more comments? > Reset peer state and config when refresh replication source failed > -- > > Key: HBASE-19923 > URL: https://issues.apache.org/jira/browse/HBASE-19923 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 3.0.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-19923.master.001.patch, > HBASE-19923.master.001.patch, HBASE-19923.master.002.patch, > HBASE-19923.master.003.patch > > > Now we use procedure for replication. When peer state changed, the RS will > read peer state from storage to cache. If RS found the peer state changed, > then it will refresh replication source. If refresh failed, the Master will > retry the procedure. Then RS will read peer state again, but now the peer > state in cache is right. So it don't refresh replication source.. So we > need reset the peer state to old peer state when refresh failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19923) Reset peer state and config when refresh replication source failed
[ https://issues.apache.org/jira/browse/HBASE-19923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358146#comment-16358146 ] Duo Zhang commented on HBASE-19923: --- No. +1. > Reset peer state and config when refresh replication source failed > -- > > Key: HBASE-19923 > URL: https://issues.apache.org/jira/browse/HBASE-19923 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 3.0.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-19923.master.001.patch, > HBASE-19923.master.001.patch, HBASE-19923.master.002.patch, > HBASE-19923.master.003.patch > > > Now we use procedure for replication. When peer state changed, the RS will > read peer state from storage to cache. If RS found the peer state changed, > then it will refresh replication source. If refresh failed, the Master will > retry the procedure. Then RS will read peer state again, but now the peer > state in cache is right. So it don't refresh replication source.. So we > need reset the peer state to old peer state when refresh failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19964) TestWriteHeavyIncrementObserver fails
[ https://issues.apache.org/jira/browse/HBASE-19964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19964: -- Attachment: HBASE-19964.patch > TestWriteHeavyIncrementObserver fails > - > > Key: HBASE-19964 > URL: https://issues.apache.org/jira/browse/HBASE-19964 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Reporter: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19964.patch > > > https://builds.apache.org/job/HBASE-Flaky-Tests/25704/artifact/hbase-examples/target/surefire-reports/org.apache.hadoop.hbase.coprocessor.example.TestWriteHeavyIncrementObserver-output.txt/*view*/ > I could reproduce it locally. Let me dig more. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19964) TestWriteHeavyIncrementObserver fails
[ https://issues.apache.org/jira/browse/HBASE-19964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19964: -- Component/s: (was: Coprocessors) test > TestWriteHeavyIncrementObserver fails > - > > Key: HBASE-19964 > URL: https://issues.apache.org/jira/browse/HBASE-19964 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19964.patch > > > https://builds.apache.org/job/HBASE-Flaky-Tests/25704/artifact/hbase-examples/target/surefire-reports/org.apache.hadoop.hbase.coprocessor.example.TestWriteHeavyIncrementObserver-output.txt/*view*/ > I could reproduce it locally. Let me dig more. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19964) TestWriteHeavyIncrementObserver fails
[ https://issues.apache.org/jira/browse/HBASE-19964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19964: -- Assignee: Duo Zhang Status: Patch Available (was: Open) > TestWriteHeavyIncrementObserver fails > - > > Key: HBASE-19964 > URL: https://issues.apache.org/jira/browse/HBASE-19964 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19964.patch > > > https://builds.apache.org/job/HBASE-Flaky-Tests/25704/artifact/hbase-examples/target/surefire-reports/org.apache.hadoop.hbase.coprocessor.example.TestWriteHeavyIncrementObserver-output.txt/*view*/ > I could reproduce it locally. Let me dig more. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19957) General framework to transit sync replication state
[ https://issues.apache.org/jira/browse/HBASE-19957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358154#comment-16358154 ] Hadoop QA commented on HBASE-19957: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 1s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} HBASE-19064 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 32s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 26s{color} | {color:green} HBASE-19064 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 27s{color} | {color:green} HBASE-19064 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 7s{color} | {color:green} HBASE-19064 passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 7m 52s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s{color} | {color:green} HBASE-19064 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 0s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 10s{color} | {color:red} hbase-replication: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 8s{color} | {color:red} hbase-server: The patch generated 2 new + 346 unchanged - 0 fixed = 348 total (was 346) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 43s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 18m 8s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m 36s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 11s{color} | {color:red} hbase-replication generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 29s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 15s{color} | {color:green} hbase-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 50s{color} | {color:green} hbase-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 18s{color} | {color:green} hbase-replication in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red}100m 38s{color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m 25s{color} | {color:green} The patch does not generate ASF Lic
[jira] [Commented] (HBASE-19965) Fix flaky TestAsyncRegionAdminApi
[ https://issues.apache.org/jira/browse/HBASE-19965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358156#comment-16358156 ] Guanghao Zhang commented on HBASE-19965: When split a region, the parent region was marked as State.SPLIT. And the daughter regions were marked as State.SPLITTING_NEW. But for merge regions, only the new child region was marked State.MERGED. And don't change the parent regions state. {code:java} public void markRegionAsSplit(final RegionInfo parent, final ServerName serverName, final RegionInfo daughterA, final RegionInfo daughterB) throws IOException { // Update hbase:meta. Parent will be marked offline and split up in hbase:meta. // The parent stays in regionStates until cleared when removed by CatalogJanitor. // Update its state in regionStates to it shows as offline and split when read // later figuring what regions are in a table and what are not: see // regionStates#getRegionsOfTable final RegionStateNode node = regionStates.getOrCreateRegionStateNode(parent); node.setState(State.SPLIT); final RegionStateNode nodeA = regionStates.getOrCreateRegionStateNode(daughterA); nodeA.setState(State.SPLITTING_NEW); final RegionStateNode nodeB = regionStates.getOrCreateRegionStateNode(daughterB); nodeB.setState(State.SPLITTING_NEW); regionStateStore.splitRegion(parent, daughterA, daughterB, serverName); if (shouldAssignFavoredNodes(parent)) { List onlineServers = this.master.getServerManager().getOnlineServersList(); ((FavoredNodesPromoter)getBalancer()). generateFavoredNodesForDaughter(onlineServers, parent, daughterA, daughterB); } } /** * When called here, the merge has happened. The two merged regions have been * unassigned and the above markRegionClosed has been called on each so they have been * disassociated from a hosting Server. The merged region will be open after this call. The * merged regions are removed from hbase:meta below> Later they are deleted from the filesystem * by the catalog janitor running against hbase:meta. It notices when the merged region no * longer holds references to the old regions. */ public void markRegionAsMerged(final RegionInfo child, final ServerName serverName, final RegionInfo mother, final RegionInfo father) throws IOException { final RegionStateNode node = regionStates.getOrCreateRegionStateNode(child); node.setState(State.MERGED); regionStates.deleteRegion(mother); regionStates.deleteRegion(father); regionStateStore.mergeRegions(child, mother, father, serverName); if (shouldAssignFavoredNodes(child)) { ((FavoredNodesPromoter)getBalancer()). generateFavoredNodesForMergedRegion(child, mother, father); } } {code} > Fix flaky TestAsyncRegionAdminApi > - > > Key: HBASE-19965 > URL: https://issues.apache.org/jira/browse/HBASE-19965 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Priority: Major > > See > [https://builds.apache.org/job/HBase%20Nightly/job/branch-2/284/testReport/junit/org.apache.hadoop.hbase.client/TestAsyncRegionAdminApi/testMergeRegions_0_/] > > java.lang.AssertionError: expected:<2> but was:<3> at > org.apache.hadoop.hbase.client.TestAsyncRegionAdminApi.testMergeRegions(TestAsyncRegionAdminApi.java:359) > > Merge regions not work. The table still have 3 regions after the > MergeRegionsProcedure finished. > The master start balance region 9e2773ba1efba79a2defa276e9a26ed4. But because > the MergeRegionsProcedure pid=138 start work first, so the balance need wait > for the lock. But after merge regions finished, the MoveRegionProcedure > pid=139 start work and assign 9e2773ba1efba79a2defa276e9a26ed4 to a new > region server. This is not right. The MoveRegionProcedure should skip to > assign a region which was marked as offline. Or we should clear the merged > regions' procedure when MergeRegionsProcedure finished. > > Logs: > 2018-02-08 16:24:44,608 INFO [master/cd4730e3eae2:0.Chore.1] > master.HMaster(1454): balance > hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., > source=cd4730e3eae2,39077,1518106776411, > destination=cd4730e3eae2,40578,1518106776318 > 2018-02-08 16:24:44,608 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37885] > procedure2.ProcedureExecutor(868): Stored pid=138, > state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE; MergeTableRegionsProcedure > table=testMergeRegions, regions=[9e2773ba1efba79a2defa276e9a26ed4, > 8f8fd5cd032313e1aadb83e31e1b7479], forcibly=false > .. > 2018-02-08 16:24:50,111 INFO [PEWorker-13] > procedure2.ProcedureExecutor(1249): Finished pid=138, state=SUCCESS; > MergeTableRegionsProcedure table=testMergeRegions, > regions=[9e2773ba1efba79a
[jira] [Commented] (HBASE-19964) TestWriteHeavyIncrementObserver fails
[ https://issues.apache.org/jira/browse/HBASE-19964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358190#comment-16358190 ] Hadoop QA commented on HBASE-19964: --- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 8s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 35s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 48s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 37s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 18m 7s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 17s{color} | {color:green} hbase-examples in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 9s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 34m 51s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 | | JIRA Issue | HBASE-19964 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12909916/HBASE-19964.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 232d0ccfcea6 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 15:49:21 UTC 2017 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / bc720374c3 | | maven | version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) | | Default Java | 1.8.0_151 | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/11465/testReport/ | | Max. process+thread count | 2387 (vs. ulimit of 1) | | modules | C: hbase-examples U: hbase-examples | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/11465/console | | Powered by | Apache Yetus 0.7.0 http://yetus.apache.org | This message was automatically generated. > TestWriteHeavyIncrementObserver fails > - > > Key: HBASE-19964 > URL: https://issues
[jira] [Commented] (HBASE-19964) TestWriteHeavyIncrementObserver fails
[ https://issues.apache.org/jira/browse/HBASE-19964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358192#comment-16358192 ] Duo Zhang commented on HBASE-19964: --- Let me commit. > TestWriteHeavyIncrementObserver fails > - > > Key: HBASE-19964 > URL: https://issues.apache.org/jira/browse/HBASE-19964 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19964.patch > > > https://builds.apache.org/job/HBASE-Flaky-Tests/25704/artifact/hbase-examples/target/surefire-reports/org.apache.hadoop.hbase.coprocessor.example.TestWriteHeavyIncrementObserver-output.txt/*view*/ > I could reproduce it locally. Let me dig more. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19964) TestWriteHeavyIncrementObserver fails
[ https://issues.apache.org/jira/browse/HBASE-19964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358203#comment-16358203 ] Duo Zhang commented on HBASE-19964: --- Pushed to master and branch-2. Keep it open for a while to see if it really works. > TestWriteHeavyIncrementObserver fails > - > > Key: HBASE-19964 > URL: https://issues.apache.org/jira/browse/HBASE-19964 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19964.patch > > > https://builds.apache.org/job/HBASE-Flaky-Tests/25704/artifact/hbase-examples/target/surefire-reports/org.apache.hadoop.hbase.coprocessor.example.TestWriteHeavyIncrementObserver-output.txt/*view*/ > I could reproduce it locally. Let me dig more. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19957) General framework to transit sync replication state
[ https://issues.apache.org/jira/browse/HBASE-19957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19957: -- Attachment: HBASE-19957-HBASE-19064-v1.patch > General framework to transit sync replication state > --- > > Key: HBASE-19957 > URL: https://issues.apache.org/jira/browse/HBASE-19957 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-19957-HBASE-19064-v1.patch, > HBASE-19957-HBASE-19064.patch, HBASE-19957.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19957) General framework to transit sync replication state
[ https://issues.apache.org/jira/browse/HBASE-19957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358217#comment-16358217 ] Duo Zhang commented on HBASE-19957: --- Address the comments on rb. And also fix the issues found by pre commit. > General framework to transit sync replication state > --- > > Key: HBASE-19957 > URL: https://issues.apache.org/jira/browse/HBASE-19957 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-19957-HBASE-19064-v1.patch, > HBASE-19957-HBASE-19064.patch, HBASE-19957.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19965) Fix flaky TestAsyncRegionAdminApi
[ https://issues.apache.org/jira/browse/HBASE-19965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358221#comment-16358221 ] Guanghao Zhang commented on HBASE-19965: The impl seems not same with the javadoc.. {code:java} public enum State { OFFLINE,// region is in an offline state OPENING,// server has begun to open but not yet done OPEN, // server opened region and updated meta CLOSING,// server has begun to close but not yet done CLOSED, // server closed region and updated meta SPLITTING, // server started split of a region SPLIT, // server completed split of a region FAILED_OPEN,// failed to open, and won't retry any more FAILED_CLOSE, // failed to close, and won't retry any more MERGING,// server started merge a region MERGED, // server completed merge a region SPLITTING_NEW, // new region to be created when RS splits a parent // region but hasn't be created yet, or master doesn't // know it's already created MERGING_NEW;// new region to be created when RS merges two // daughter regions but hasn't be created yet, or // master doesn't know it's already created {code} > Fix flaky TestAsyncRegionAdminApi > - > > Key: HBASE-19965 > URL: https://issues.apache.org/jira/browse/HBASE-19965 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Priority: Major > > See > [https://builds.apache.org/job/HBase%20Nightly/job/branch-2/284/testReport/junit/org.apache.hadoop.hbase.client/TestAsyncRegionAdminApi/testMergeRegions_0_/] > > java.lang.AssertionError: expected:<2> but was:<3> at > org.apache.hadoop.hbase.client.TestAsyncRegionAdminApi.testMergeRegions(TestAsyncRegionAdminApi.java:359) > > Merge regions not work. The table still have 3 regions after the > MergeRegionsProcedure finished. > The master start balance region 9e2773ba1efba79a2defa276e9a26ed4. But because > the MergeRegionsProcedure pid=138 start work first, so the balance need wait > for the lock. But after merge regions finished, the MoveRegionProcedure > pid=139 start work and assign 9e2773ba1efba79a2defa276e9a26ed4 to a new > region server. This is not right. The MoveRegionProcedure should skip to > assign a region which was marked as offline. Or we should clear the merged > regions' procedure when MergeRegionsProcedure finished. > > Logs: > 2018-02-08 16:24:44,608 INFO [master/cd4730e3eae2:0.Chore.1] > master.HMaster(1454): balance > hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., > source=cd4730e3eae2,39077,1518106776411, > destination=cd4730e3eae2,40578,1518106776318 > 2018-02-08 16:24:44,608 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37885] > procedure2.ProcedureExecutor(868): Stored pid=138, > state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE; MergeTableRegionsProcedure > table=testMergeRegions, regions=[9e2773ba1efba79a2defa276e9a26ed4, > 8f8fd5cd032313e1aadb83e31e1b7479], forcibly=false > .. > 2018-02-08 16:24:50,111 INFO [PEWorker-13] > procedure2.ProcedureExecutor(1249): Finished pid=138, state=SUCCESS; > MergeTableRegionsProcedure table=testMergeRegions, > regions=[9e2773ba1efba79a2defa276e9a26ed4, 8f8fd5cd032313e1aadb83e31e1b7479], > forcibly=false in 5.5710sec > 2018-02-08 16:24:50,113 INFO [PEWorker-13] > procedure.MasterProcedureScheduler(813): pid=139, > state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure > hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., > source=cd4730e3eae2,39077,1518106776411, > destination=cd4730e3eae2,40578,1518106776318 testMergeRegions > testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19923) Reset peer state and config when refresh replication source failed
[ https://issues.apache.org/jira/browse/HBASE-19923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358279#comment-16358279 ] Guanghao Zhang commented on HBASE-19923: Pushed to master and HBASE-19397-branch-2. Thanks [~Apache9] for reviewing. > Reset peer state and config when refresh replication source failed > -- > > Key: HBASE-19923 > URL: https://issues.apache.org/jira/browse/HBASE-19923 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 3.0.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-19923.master.001.patch, > HBASE-19923.master.001.patch, HBASE-19923.master.002.patch, > HBASE-19923.master.003.patch > > > Now we use procedure for replication. When peer state changed, the RS will > read peer state from storage to cache. If RS found the peer state changed, > then it will refresh replication source. If refresh failed, the Master will > retry the procedure. Then RS will read peer state again, but now the peer > state in cache is right. So it don't refresh replication source.. So we > need reset the peer state to old peer state when refresh failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19554) AbstractTestDLS.testThreeRSAbort sometimes fails in pre commit
[ https://issues.apache.org/jira/browse/HBASE-19554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358324#comment-16358324 ] Duo Zhang commented on HBASE-19554: --- I guess the problem is related to meta reassign. [~zghaobac] has already found something strange for AMv2 in HBASE-19965, so wait for his digging result. And I found something strange in the output {noformat} java.lang.AssertionError at org.apache.hadoop.hbase.wal.WALKeyImpl.getWriteEntry(WALKeyImpl.java:82) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:159) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeMarker(WALUtil.java:132) at org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeRegionEventMarker(WALUtil.java:97) at org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1103) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1615) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {noformat} The assertion is {code} public MultiVersionConcurrencyControl.WriteEntry getWriteEntry() throws InterruptedIOException { assert this.writeEntry != null; return this.writeEntry; } {code} I think the problem is introduced by HBASE-19929. Since we have closed WAL directly, then it is possible that we fail a WAL.append without assigning a mvcc writeEntry. Let me file a issue to fix this. > AbstractTestDLS.testThreeRSAbort sometimes fails in pre commit > -- > > Key: HBASE-19554 > URL: https://issues.apache.org/jira/browse/HBASE-19554 > Project: HBase > Issue Type: Sub-task > Components: Recovery, wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19554.patch > > > https://builds.apache.org/job/PreCommit-HBASE-Build/10554/artifact/patchprocess/patch-unit-hbase-server.txt > The error message is a bit strange: > {quote} > [ERROR] testThreeRSAbort(org.apache.hadoop.hbase.master.TestDLSAsyncFSWAL) > Time elapsed: 20.627 s <<< ERROR! > org.apache.hadoop.hbase.TableNotFoundException: Region of > 'hbase:namespace,,1513320505933.451650152885a3b41d0b1110deca513c.' is > expected in the table of 'testThreeRSAbort', but hbase:meta says it is in the > table of 'hbase:namespace'. hbase:meta might be damaged. > {quote} > It fails for both FSHLog and AsyncFSWAL. Need to dig more. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-19966) The WriteEntry for WALKey maybe null if we failed to call WAL.append
Duo Zhang created HBASE-19966: - Summary: The WriteEntry for WALKey maybe null if we failed to call WAL.append Key: HBASE-19966 URL: https://issues.apache.org/jira/browse/HBASE-19966 Project: HBase Issue Type: Bug Components: wal Reporter: Duo Zhang Assignee: Duo Zhang Fix For: 2.0.0-beta-2 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19966) The WriteEntry for WALKey maybe null if we failed to call WAL.append
[ https://issues.apache.org/jira/browse/HBASE-19966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19966: -- Attachment: HBASE-19966.patch > The WriteEntry for WALKey maybe null if we failed to call WAL.append > > > Key: HBASE-19966 > URL: https://issues.apache.org/jira/browse/HBASE-19966 > Project: HBase > Issue Type: Bug > Components: wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19966.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19966) The WriteEntry for WALKey maybe null if we failed to call WAL.append
[ https://issues.apache.org/jira/browse/HBASE-19966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19966: -- Status: Patch Available (was: Open) > The WriteEntry for WALKey maybe null if we failed to call WAL.append > > > Key: HBASE-19966 > URL: https://issues.apache.org/jira/browse/HBASE-19966 > Project: HBase > Issue Type: Bug > Components: wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19966.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19966) The WriteEntry for WALKey maybe null if we failed to call WAL.append
[ https://issues.apache.org/jira/browse/HBASE-19966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358349#comment-16358349 ] Duo Zhang commented on HBASE-19966: --- Remove the assertion and add a null check in WALUtil.doFullAppendTransaction. Add a test to confirm that the WriteEntry could be null if we append on a closed WAL. Also cleanup some warnings for the tests. > The WriteEntry for WALKey maybe null if we failed to call WAL.append > > > Key: HBASE-19966 > URL: https://issues.apache.org/jira/browse/HBASE-19966 > Project: HBase > Issue Type: Bug > Components: wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19966.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19957) General framework to transit sync replication state
[ https://issues.apache.org/jira/browse/HBASE-19957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358366#comment-16358366 ] Hadoop QA commented on HBASE-19957: --- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} HBASE-19064 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 24s{color} | {color:green} HBASE-19064 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 14s{color} | {color:green} HBASE-19064 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 26s{color} | {color:green} HBASE-19064 passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 7m 5s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s{color} | {color:green} HBASE-19064 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 41s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 18m 7s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 25s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 30s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 15s{color} | {color:green} hbase-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 57s{color} | {color:green} hbase-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s{color} | {color:green} hbase-replication in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green}109m 33s{color} | {color:green} hbase-server in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 2m 18s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}164m 22s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 | | JIRA Issue | HBASE-19957 | | JIRA Patch URL | https://iss
[jira] [Commented] (HBASE-19957) General framework to transit sync replication state
[ https://issues.apache.org/jira/browse/HBASE-19957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358373#comment-16358373 ] Duo Zhang commented on HBASE-19957: --- Any other concerns? [~zghaobac]? If no let's commit this and start working on the follow-on issues. This one has been blocking us for a long time... Thanks. > General framework to transit sync replication state > --- > > Key: HBASE-19957 > URL: https://issues.apache.org/jira/browse/HBASE-19957 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-19957-HBASE-19064-v1.patch, > HBASE-19957-HBASE-19064.patch, HBASE-19957.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19942) Fix flaky TestSimpleRpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-19942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358397#comment-16358397 ] Hudson commented on HBASE-19942: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4554 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4554/]) HBASE-19942 (addendum) Fix flaky TestSimpleRpcScheduler (zghao: rev 0a5f26324cfe4eb39eb48947f11364ec39221fa6) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestSimpleRpcScheduler.java > Fix flaky TestSimpleRpcScheduler > > > Key: HBASE-19942 > URL: https://issues.apache.org/jira/browse/HBASE-19942 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19942.branch-2.001.patch, > HBASE-19942.branch-2.addendum.patch, HBASE-19942.master.001.patch, > HBASE-19942.master.addendum.patch > > > [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests-branch2.0/lastSuccessfulBuild/artifact/dashboard.html] > > https://builds.apache.org/job/HBASE-Flaky-Tests-branch2.0/1387/testReport/junit/org.apache.hadoop.hbase.ipc/TestSimpleRpcScheduler/testSoftAndHardQueueLimits/ > > h3. Stacktrace > java.lang.AssertionError at > org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler.testSoftAndHardQueueLimits(TestSimpleRpcScheduler.java:451) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19937) Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer are initialized
[ https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358398#comment-16358398 ] Hudson commented on HBASE-19937: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4554 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4554/]) HBASE-19937 Ensure createRSGroupTable be called after ProcedureExecutor (tedyu: rev bc720374c33701c13f6227c0edfe3a792e7aa333) * (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java * (add) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestEnableRSGroup.java * (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java * (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java > Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer > are initialized > - > > Key: HBASE-19937 > URL: https://issues.apache.org/jira/browse/HBASE-19937 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.0.0-beta-2 >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19937.branch-2.001.patch, > HBASE-19937.branch-2.002.patch, HBASE-19937.branch-2.003.patch, > HBASE-19937.branch-2.004.patch, HBASE-19937.branch-2.005.patch, > HBASE-19937.branch-2.006.patch, import-order.png > > > hbase:rsgroup table will be created by calling createRSGroupTable when master > load system coprocessors in > > {code:java} > 844 this.cpHost = new MasterCoprocessorHost(this, this.conf);{code} > when ProcedureExecutor hasn't been initialized before createRSGroupTable, it > will encounter Exception as follows, > > {noformat} > Exception in thread > "org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130" > java.lang.IllegalArgumentException > at > org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835) > at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795) > at > org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858) > at > org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823) > at > org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat} > while ProcedureExecutor initialized by calling > {code:java} > 848 startServiceThreads();{code} > And LoadBalancer is initialized by calling > {code:java} > 868 this.balancer.initialize();{code} > When LoadBalancer hasn't been initialized before createRSGroupTable, it will > encounters Exception as follows, > 2018-02-02,16:12:45,688 ERROR > org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught > runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; > CreateTableProcedure table=hbase:rsgroup > java.lang.NullPointerException > at > org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254) > at > org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162) > at > org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603) > at > org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108) > at > org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51) > at > org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182) > at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738) > > As a result of CreateTableProcedure.rollbackState, it may then print logs > warning TableExistsException as follows, > 2018-02-02,16:12:55,503 WARN > org.apache.hadoop.hbase.rsgroup.RSGroupInf
[jira] [Commented] (HBASE-19957) General framework to transit sync replication state
[ https://issues.apache.org/jira/browse/HBASE-19957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358406#comment-16358406 ] Guanghao Zhang commented on HBASE-19957: +1 > General framework to transit sync replication state > --- > > Key: HBASE-19957 > URL: https://issues.apache.org/jira/browse/HBASE-19957 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-19957-HBASE-19064-v1.patch, > HBASE-19957-HBASE-19064.patch, HBASE-19957.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19957) General framework to transit sync replication state
[ https://issues.apache.org/jira/browse/HBASE-19957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19957: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HBASE-19064 Status: Resolved (was: Patch Available) Rebased and pushed to branch HBASE-19064. Thanks [~zghaobac] for reviewing. > General framework to transit sync replication state > --- > > Key: HBASE-19957 > URL: https://issues.apache.org/jira/browse/HBASE-19957 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: HBASE-19064 > > Attachments: HBASE-19957-HBASE-19064-v1.patch, > HBASE-19957-HBASE-19064.patch, HBASE-19957.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19082) Reject read/write from client but accept write from replication in state S
[ https://issues.apache.org/jira/browse/HBASE-19082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19082: -- Summary: Reject read/write from client but accept write from replication in state S (was: Implement a procedure to convert RS from DA to S) > Reject read/write from client but accept write from replication in state S > -- > > Key: HBASE-19082 > URL: https://issues.apache.org/jira/browse/HBASE-19082 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Duo Zhang >Priority: Major > Labels: HBASE-19064 > Fix For: 3.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19082) Reject read/write from client but accept write from replication in state S
[ https://issues.apache.org/jira/browse/HBASE-19082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358482#comment-16358482 ] Duo Zhang commented on HBASE-19082: --- We use the same procedure for transiting so changed the title to be more specific. > Reject read/write from client but accept write from replication in state S > -- > > Key: HBASE-19082 > URL: https://issues.apache.org/jira/browse/HBASE-19082 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Duo Zhang >Priority: Major > Labels: HBASE-19064 > Fix For: 3.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19966) The WriteEntry for WALKey maybe null if we failed to call WAL.append
[ https://issues.apache.org/jira/browse/HBASE-19966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358553#comment-16358553 ] Hadoop QA commented on HBASE-19966: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 32s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 10s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m 24s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 11s{color} | {color:red} hbase-server: The patch generated 3 new + 13 unchanged - 6 fixed = 16 total (was 19) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m 4s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 20m 2s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}105m 20s{color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 20s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}146m 59s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hbase.replication.TestReplicationDroppedTables | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 | | JIRA Issue | HBASE-19966 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12909936/HBASE-19966.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 35b1b7260894 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 2b63af376e | | maven | version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) | | Default Java | 1.8.0_151 | | checkstyle | https://builds.apache.org/job/PreCommit-HBASE-Build/11467/artifact/patchprocess/diff-checkstyle-hbase-server.txt | | unit | https://builds.apache.org/job/PreCommit-HBASE-Build/11467/artifact/patchprocess/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/11467/testReport/ | | Max. process+thread count | 4704 (vs. ulimit of 10
[jira] [Resolved] (HBASE-4128) Detect whether there was zookeeper ensemble, master or region server hanging from previous build
[ https://issues.apache.org/jira/browse/HBASE-4128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-4128. --- Resolution: Not A Problem > Detect whether there was zookeeper ensemble, master or region server hanging > from previous build > > > Key: HBASE-4128 > URL: https://issues.apache.org/jira/browse/HBASE-4128 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Priority: Major > > Quite often, we see unit test(s) time out after 15 minutes. > One example was TestShell: > https://builds.apache.org/view/G-L/view/HBase/job/hbase-0.90/239/console > This may be caused by zookeeper ensemble, master or region server hanging > from previous build. > We should detect (and terminate, if possible) the hanging zk ensemble, master > or region server from previous build as the first step in current build. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19920) TokenUtil.obtainToken unnecessarily creates a local directory
[ https://issues.apache.org/jira/browse/HBASE-19920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-19920: -- Status: Open (was: Patch Available) > TokenUtil.obtainToken unnecessarily creates a local directory > - > > Key: HBASE-19920 > URL: https://issues.apache.org/jira/browse/HBASE-19920 > Project: HBase > Issue Type: Bug >Reporter: Rohini Palaniswamy >Assignee: Mike Drob >Priority: Major > Fix For: 2.0 > > Attachments: HBASE-19920.patch, HBASE-19920.v2.patch, > HBASE-19920.v3.patch, HBASE-19920.v4.patch, HBASE-19920.v5.patch, > HBASE-19920.v6.patch, HBASE-19920.v7.patch, HBASE-19920.v8.patch > > > On client code, when one calls TokenUtil.obtainToken it loads ProtobufUtil > which in its static block initializes DynamicClassLoader and that creates the > directory ${hbase.local.dir}/jars/ and also instantiates a filesystem class > to access hbase.dynamic.jars.dir. > https://github.com/apache/hbase/blob/master/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DynamicClassLoader.java#L109-L127 > Since this is region server specific code, not expecting this to happen when > one accesses hbase as a client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19920) TokenUtil.obtainToken unnecessarily creates a local directory
[ https://issues.apache.org/jira/browse/HBASE-19920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-19920: -- Status: Patch Available (was: Open) > TokenUtil.obtainToken unnecessarily creates a local directory > - > > Key: HBASE-19920 > URL: https://issues.apache.org/jira/browse/HBASE-19920 > Project: HBase > Issue Type: Bug >Reporter: Rohini Palaniswamy >Assignee: Mike Drob >Priority: Major > Fix For: 2.0 > > Attachments: HBASE-19920.patch, HBASE-19920.v2.patch, > HBASE-19920.v3.patch, HBASE-19920.v4.patch, HBASE-19920.v5.patch, > HBASE-19920.v6.patch, HBASE-19920.v7.patch, HBASE-19920.v8.patch > > > On client code, when one calls TokenUtil.obtainToken it loads ProtobufUtil > which in its static block initializes DynamicClassLoader and that creates the > directory ${hbase.local.dir}/jars/ and also instantiates a filesystem class > to access hbase.dynamic.jars.dir. > https://github.com/apache/hbase/blob/master/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DynamicClassLoader.java#L109-L127 > Since this is region server specific code, not expecting this to happen when > one accesses hbase as a client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-19965) Fix flaky TestAsyncRegionAdminApi
[ https://issues.apache.org/jira/browse/HBASE-19965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack reassigned HBASE-19965: - Assignee: stack > Fix flaky TestAsyncRegionAdminApi > - > > Key: HBASE-19965 > URL: https://issues.apache.org/jira/browse/HBASE-19965 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: stack >Priority: Major > > See > [https://builds.apache.org/job/HBase%20Nightly/job/branch-2/284/testReport/junit/org.apache.hadoop.hbase.client/TestAsyncRegionAdminApi/testMergeRegions_0_/] > > java.lang.AssertionError: expected:<2> but was:<3> at > org.apache.hadoop.hbase.client.TestAsyncRegionAdminApi.testMergeRegions(TestAsyncRegionAdminApi.java:359) > > Merge regions not work. The table still have 3 regions after the > MergeRegionsProcedure finished. > The master start balance region 9e2773ba1efba79a2defa276e9a26ed4. But because > the MergeRegionsProcedure pid=138 start work first, so the balance need wait > for the lock. But after merge regions finished, the MoveRegionProcedure > pid=139 start work and assign 9e2773ba1efba79a2defa276e9a26ed4 to a new > region server. This is not right. The MoveRegionProcedure should skip to > assign a region which was marked as offline. Or we should clear the merged > regions' procedure when MergeRegionsProcedure finished. > > Logs: > 2018-02-08 16:24:44,608 INFO [master/cd4730e3eae2:0.Chore.1] > master.HMaster(1454): balance > hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., > source=cd4730e3eae2,39077,1518106776411, > destination=cd4730e3eae2,40578,1518106776318 > 2018-02-08 16:24:44,608 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37885] > procedure2.ProcedureExecutor(868): Stored pid=138, > state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE; MergeTableRegionsProcedure > table=testMergeRegions, regions=[9e2773ba1efba79a2defa276e9a26ed4, > 8f8fd5cd032313e1aadb83e31e1b7479], forcibly=false > .. > 2018-02-08 16:24:50,111 INFO [PEWorker-13] > procedure2.ProcedureExecutor(1249): Finished pid=138, state=SUCCESS; > MergeTableRegionsProcedure table=testMergeRegions, > regions=[9e2773ba1efba79a2defa276e9a26ed4, 8f8fd5cd032313e1aadb83e31e1b7479], > forcibly=false in 5.5710sec > 2018-02-08 16:24:50,113 INFO [PEWorker-13] > procedure.MasterProcedureScheduler(813): pid=139, > state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure > hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., > source=cd4730e3eae2,39077,1518106776411, > destination=cd4730e3eae2,40578,1518106776318 testMergeRegions > testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19965) Fix flaky TestAsyncRegionAdminApi
[ https://issues.apache.org/jira/browse/HBASE-19965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358602#comment-16358602 ] stack commented on HBASE-19965: --- Thank you for the research [~zghaobac]; let me have a go at it boss. Nice work sir. > Fix flaky TestAsyncRegionAdminApi > - > > Key: HBASE-19965 > URL: https://issues.apache.org/jira/browse/HBASE-19965 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: stack >Priority: Major > Fix For: 2.0.0-beta-2 > > > See > [https://builds.apache.org/job/HBase%20Nightly/job/branch-2/284/testReport/junit/org.apache.hadoop.hbase.client/TestAsyncRegionAdminApi/testMergeRegions_0_/] > > java.lang.AssertionError: expected:<2> but was:<3> at > org.apache.hadoop.hbase.client.TestAsyncRegionAdminApi.testMergeRegions(TestAsyncRegionAdminApi.java:359) > > Merge regions not work. The table still have 3 regions after the > MergeRegionsProcedure finished. > The master start balance region 9e2773ba1efba79a2defa276e9a26ed4. But because > the MergeRegionsProcedure pid=138 start work first, so the balance need wait > for the lock. But after merge regions finished, the MoveRegionProcedure > pid=139 start work and assign 9e2773ba1efba79a2defa276e9a26ed4 to a new > region server. This is not right. The MoveRegionProcedure should skip to > assign a region which was marked as offline. Or we should clear the merged > regions' procedure when MergeRegionsProcedure finished. > > Logs: > 2018-02-08 16:24:44,608 INFO [master/cd4730e3eae2:0.Chore.1] > master.HMaster(1454): balance > hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., > source=cd4730e3eae2,39077,1518106776411, > destination=cd4730e3eae2,40578,1518106776318 > 2018-02-08 16:24:44,608 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37885] > procedure2.ProcedureExecutor(868): Stored pid=138, > state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE; MergeTableRegionsProcedure > table=testMergeRegions, regions=[9e2773ba1efba79a2defa276e9a26ed4, > 8f8fd5cd032313e1aadb83e31e1b7479], forcibly=false > .. > 2018-02-08 16:24:50,111 INFO [PEWorker-13] > procedure2.ProcedureExecutor(1249): Finished pid=138, state=SUCCESS; > MergeTableRegionsProcedure table=testMergeRegions, > regions=[9e2773ba1efba79a2defa276e9a26ed4, 8f8fd5cd032313e1aadb83e31e1b7479], > forcibly=false in 5.5710sec > 2018-02-08 16:24:50,113 INFO [PEWorker-13] > procedure.MasterProcedureScheduler(813): pid=139, > state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure > hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., > source=cd4730e3eae2,39077,1518106776411, > destination=cd4730e3eae2,40578,1518106776318 testMergeRegions > testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19965) Fix flaky TestAsyncRegionAdminApi
[ https://issues.apache.org/jira/browse/HBASE-19965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19965: -- Priority: Critical (was: Major) > Fix flaky TestAsyncRegionAdminApi > - > > Key: HBASE-19965 > URL: https://issues.apache.org/jira/browse/HBASE-19965 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-2 > > > See > [https://builds.apache.org/job/HBase%20Nightly/job/branch-2/284/testReport/junit/org.apache.hadoop.hbase.client/TestAsyncRegionAdminApi/testMergeRegions_0_/] > > java.lang.AssertionError: expected:<2> but was:<3> at > org.apache.hadoop.hbase.client.TestAsyncRegionAdminApi.testMergeRegions(TestAsyncRegionAdminApi.java:359) > > Merge regions not work. The table still have 3 regions after the > MergeRegionsProcedure finished. > The master start balance region 9e2773ba1efba79a2defa276e9a26ed4. But because > the MergeRegionsProcedure pid=138 start work first, so the balance need wait > for the lock. But after merge regions finished, the MoveRegionProcedure > pid=139 start work and assign 9e2773ba1efba79a2defa276e9a26ed4 to a new > region server. This is not right. The MoveRegionProcedure should skip to > assign a region which was marked as offline. Or we should clear the merged > regions' procedure when MergeRegionsProcedure finished. > > Logs: > 2018-02-08 16:24:44,608 INFO [master/cd4730e3eae2:0.Chore.1] > master.HMaster(1454): balance > hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., > source=cd4730e3eae2,39077,1518106776411, > destination=cd4730e3eae2,40578,1518106776318 > 2018-02-08 16:24:44,608 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37885] > procedure2.ProcedureExecutor(868): Stored pid=138, > state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE; MergeTableRegionsProcedure > table=testMergeRegions, regions=[9e2773ba1efba79a2defa276e9a26ed4, > 8f8fd5cd032313e1aadb83e31e1b7479], forcibly=false > .. > 2018-02-08 16:24:50,111 INFO [PEWorker-13] > procedure2.ProcedureExecutor(1249): Finished pid=138, state=SUCCESS; > MergeTableRegionsProcedure table=testMergeRegions, > regions=[9e2773ba1efba79a2defa276e9a26ed4, 8f8fd5cd032313e1aadb83e31e1b7479], > forcibly=false in 5.5710sec > 2018-02-08 16:24:50,113 INFO [PEWorker-13] > procedure.MasterProcedureScheduler(813): pid=139, > state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure > hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., > source=cd4730e3eae2,39077,1518106776411, > destination=cd4730e3eae2,40578,1518106776318 testMergeRegions > testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19965) Fix flaky TestAsyncRegionAdminApi
[ https://issues.apache.org/jira/browse/HBASE-19965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19965: -- Fix Version/s: 2.0.0-beta-2 > Fix flaky TestAsyncRegionAdminApi > - > > Key: HBASE-19965 > URL: https://issues.apache.org/jira/browse/HBASE-19965 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: stack >Priority: Major > Fix For: 2.0.0-beta-2 > > > See > [https://builds.apache.org/job/HBase%20Nightly/job/branch-2/284/testReport/junit/org.apache.hadoop.hbase.client/TestAsyncRegionAdminApi/testMergeRegions_0_/] > > java.lang.AssertionError: expected:<2> but was:<3> at > org.apache.hadoop.hbase.client.TestAsyncRegionAdminApi.testMergeRegions(TestAsyncRegionAdminApi.java:359) > > Merge regions not work. The table still have 3 regions after the > MergeRegionsProcedure finished. > The master start balance region 9e2773ba1efba79a2defa276e9a26ed4. But because > the MergeRegionsProcedure pid=138 start work first, so the balance need wait > for the lock. But after merge regions finished, the MoveRegionProcedure > pid=139 start work and assign 9e2773ba1efba79a2defa276e9a26ed4 to a new > region server. This is not right. The MoveRegionProcedure should skip to > assign a region which was marked as offline. Or we should clear the merged > regions' procedure when MergeRegionsProcedure finished. > > Logs: > 2018-02-08 16:24:44,608 INFO [master/cd4730e3eae2:0.Chore.1] > master.HMaster(1454): balance > hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., > source=cd4730e3eae2,39077,1518106776411, > destination=cd4730e3eae2,40578,1518106776318 > 2018-02-08 16:24:44,608 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37885] > procedure2.ProcedureExecutor(868): Stored pid=138, > state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE; MergeTableRegionsProcedure > table=testMergeRegions, regions=[9e2773ba1efba79a2defa276e9a26ed4, > 8f8fd5cd032313e1aadb83e31e1b7479], forcibly=false > .. > 2018-02-08 16:24:50,111 INFO [PEWorker-13] > procedure2.ProcedureExecutor(1249): Finished pid=138, state=SUCCESS; > MergeTableRegionsProcedure table=testMergeRegions, > regions=[9e2773ba1efba79a2defa276e9a26ed4, 8f8fd5cd032313e1aadb83e31e1b7479], > forcibly=false in 5.5710sec > 2018-02-08 16:24:50,113 INFO [PEWorker-13] > procedure.MasterProcedureScheduler(813): pid=139, > state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure > hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., > source=cd4730e3eae2,39077,1518106776411, > destination=cd4730e3eae2,40578,1518106776318 testMergeRegions > testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19876) The exception happening in converting pb mutation to hbase.mutation messes up the CellScanner
[ https://issues.apache.org/jira/browse/HBASE-19876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-19876: --- Attachment: HBASE-19876.branch-1.2.v0.patch > The exception happening in converting pb mutation to hbase.mutation messes up > the CellScanner > - > > Key: HBASE-19876 > URL: https://issues.apache.org/jira/browse/HBASE-19876 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 1.3.2, 1.5.0, 1.2.7, 2.0.0-beta-2, 1.4.2 > > Attachments: HBASE-19876.branch-1.2.v0.patch, > HBASE-19876.master.001.patch, HBASE-19876.v0.patch, HBASE-19876.v1.patch, > HBASE-19876.v2.patch, HBASE-19876.v3.patch, HBASE-19876.v3.patch, > HBASE-19876.v3.patch, HBASE-19876.v3.patch, HBASE-19876.v4.patch, > HBASE-19876.v5.patch > > > {code:java} > 2018-01-27 22:51:43,794 INFO [hconnection-0x3291b443-shared-pool11-t6] > client.AsyncRequestFutureImpl(778): id=5, table=testQuotaStatusFromMaster3, > attempt=6/16 failed=20ops, last > exception=org.apache.hadoop.hbase.client.WrongRowIOException: > org.apache.hadoop.hbase.client.WrongRowIOException: The row in xxx doesn't > match the original one aaa > at org.apache.hadoop.hbase.client.Mutation.add(Mutation.java:776) > at org.apache.hadoop.hbase.client.Put.add(Put.java:282) > at > org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toPut(ProtobufUtil.java:642) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:952) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:896) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2591) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304){code} > I noticed this bug when testing the table space quota. > When rs are converting pb mutation to hbase.mutation, the quota exception or > cell exception may be thrown. > {code} > Unable to find source-code formatter for language: > rsrpcservices#dobatchop.java. Available languages are: actionscript, ada, > applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, > java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, > rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml for > (ClientProtos.Action action: mutations) { > MutationProto m = action.getMutation(); > Mutation mutation; > if (m.getMutateType() == MutationType.PUT) { > mutation = ProtobufUtil.toPut(m, cells); > batchContainsPuts = true; > } else { > mutation = ProtobufUtil.toDelete(m, cells); > batchContainsDelete = true; > } > mutationActionMap.put(mutation, action); > mArray[i++] = mutation; > checkCellSizeLimit(region, mutation); > // Check if a space quota disallows this mutation > spaceQuotaEnforcement.getPolicyEnforcement(region).check(mutation); > quota.addMutation(mutation); > } > {code} > rs has caught the exception but it doesn't have the cellscanner skip the > failed cells. > {code:java} > } catch (IOException ie) { > if (atomic) { > throw ie; > } > for (Action mutation : mutations) { > builder.addResultOrException(getResultOrException(ie, > mutation.getIndex())); > } > } > {code} > The bug results in the WrongRowIOException to remaining mutations since they > refer to invalid cells. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19876) The exception happening in converting pb mutation to hbase.mutation messes up the CellScanner
[ https://issues.apache.org/jira/browse/HBASE-19876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-19876: --- Status: Patch Available (was: Open) > The exception happening in converting pb mutation to hbase.mutation messes up > the CellScanner > - > > Key: HBASE-19876 > URL: https://issues.apache.org/jira/browse/HBASE-19876 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 1.3.2, 1.5.0, 1.2.7, 2.0.0-beta-2, 1.4.2 > > Attachments: HBASE-19876.branch-1.2.v0.patch, > HBASE-19876.master.001.patch, HBASE-19876.v0.patch, HBASE-19876.v1.patch, > HBASE-19876.v2.patch, HBASE-19876.v3.patch, HBASE-19876.v3.patch, > HBASE-19876.v3.patch, HBASE-19876.v3.patch, HBASE-19876.v4.patch, > HBASE-19876.v5.patch > > > {code:java} > 2018-01-27 22:51:43,794 INFO [hconnection-0x3291b443-shared-pool11-t6] > client.AsyncRequestFutureImpl(778): id=5, table=testQuotaStatusFromMaster3, > attempt=6/16 failed=20ops, last > exception=org.apache.hadoop.hbase.client.WrongRowIOException: > org.apache.hadoop.hbase.client.WrongRowIOException: The row in xxx doesn't > match the original one aaa > at org.apache.hadoop.hbase.client.Mutation.add(Mutation.java:776) > at org.apache.hadoop.hbase.client.Put.add(Put.java:282) > at > org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toPut(ProtobufUtil.java:642) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:952) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:896) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2591) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304){code} > I noticed this bug when testing the table space quota. > When rs are converting pb mutation to hbase.mutation, the quota exception or > cell exception may be thrown. > {code} > Unable to find source-code formatter for language: > rsrpcservices#dobatchop.java. Available languages are: actionscript, ada, > applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, > java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, > rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml for > (ClientProtos.Action action: mutations) { > MutationProto m = action.getMutation(); > Mutation mutation; > if (m.getMutateType() == MutationType.PUT) { > mutation = ProtobufUtil.toPut(m, cells); > batchContainsPuts = true; > } else { > mutation = ProtobufUtil.toDelete(m, cells); > batchContainsDelete = true; > } > mutationActionMap.put(mutation, action); > mArray[i++] = mutation; > checkCellSizeLimit(region, mutation); > // Check if a space quota disallows this mutation > spaceQuotaEnforcement.getPolicyEnforcement(region).check(mutation); > quota.addMutation(mutation); > } > {code} > rs has caught the exception but it doesn't have the cellscanner skip the > failed cells. > {code:java} > } catch (IOException ie) { > if (atomic) { > throw ie; > } > for (Action mutation : mutations) { > builder.addResultOrException(getResultOrException(ie, > mutation.getIndex())); > } > } > {code} > The bug results in the WrongRowIOException to remaining mutations since they > refer to invalid cells. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19735) Create a minimal "client" tarball installation
[ https://issues.apache.org/jira/browse/HBASE-19735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358669#comment-16358669 ] Josh Elser commented on HBASE-19735: Looks like org.apache.hadoop.hbase.client.TestAsyncTableGetMultiThreaded is what failed (which I can't see how it would be related to this change). [~mdrob], [~stack]: thing this is good for master? > Create a minimal "client" tarball installation > -- > > Key: HBASE-19735 > URL: https://issues.apache.org/jira/browse/HBASE-19735 > Project: HBase > Issue Type: New Feature >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-19735.000.patch, HBASE-19735.001.branch-2.patch, > HBASE-19735.002.branch-2.patch, HBASE-19735.003.patch, HBASE-19735.004.patch > > > We're moving ourselves towards more controlled dependencies. A logical next > step is to try to do the same for our "binary" artifacts that we create > during releases. > There is code (our's and our dependency's) which the HMaster and RegionServer > require which, obviously, clients do not need. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19966) The WriteEntry for WALKey maybe null if we failed to call WAL.append
[ https://issues.apache.org/jira/browse/HBASE-19966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358708#comment-16358708 ] stack commented on HBASE-19966: --- Copy to a local variable just in case it gets set to null between check and use? mvcc.complete(walKey.getWriteEntry()); Can fix on commit. +1 (Is failed test related?) > The WriteEntry for WALKey maybe null if we failed to call WAL.append > > > Key: HBASE-19966 > URL: https://issues.apache.org/jira/browse/HBASE-19966 > Project: HBase > Issue Type: Bug > Components: wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19966.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19923) Reset peer state and config when refresh replication source failed
[ https://issues.apache.org/jira/browse/HBASE-19923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358722#comment-16358722 ] Hudson commented on HBASE-19923: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4555 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4555/]) HBASE-19923 Reset peer state and config when refresh replication source (zghao: rev 2b63af376edbca53eb0475e929478bd38b580aae) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/PeerProcedureHandlerImpl.java * (edit) hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeerImpl.java > Reset peer state and config when refresh replication source failed > -- > > Key: HBASE-19923 > URL: https://issues.apache.org/jira/browse/HBASE-19923 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 3.0.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-19923.master.001.patch, > HBASE-19923.master.001.patch, HBASE-19923.master.002.patch, > HBASE-19923.master.003.patch > > > Now we use procedure for replication. When peer state changed, the RS will > read peer state from storage to cache. If RS found the peer state changed, > then it will refresh replication source. If refresh failed, the Master will > retry the procedure. Then RS will read peer state again, but now the peer > state in cache is right. So it don't refresh replication source.. So we > need reset the peer state to old peer state when refresh failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19964) TestWriteHeavyIncrementObserver fails
[ https://issues.apache.org/jira/browse/HBASE-19964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358721#comment-16358721 ] Hudson commented on HBASE-19964: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4555 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4555/]) HBASE-19964 TestWriteHeavyIncrementObserver fails (zhangduo: rev 05279d2f5f45f170267c85e16e8472773094a59f) * (edit) hbase-examples/src/test/java/org/apache/hadoop/hbase/coprocessor/example/TestWriteHeavyIncrementObserver.java > TestWriteHeavyIncrementObserver fails > - > > Key: HBASE-19964 > URL: https://issues.apache.org/jira/browse/HBASE-19964 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19964.patch > > > https://builds.apache.org/job/HBASE-Flaky-Tests/25704/artifact/hbase-examples/target/surefire-reports/org.apache.hadoop.hbase.coprocessor.example.TestWriteHeavyIncrementObserver-output.txt/*view*/ > I could reproduce it locally. Let me dig more. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-15381) Implement a distributed MOB compaction by procedure
[ https://issues.apache.org/jira/browse/HBASE-15381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358731#comment-16358731 ] huaxiang sun commented on HBASE-15381: -- Thanks [~jingcheng...@intel.com], I am taking over. > Implement a distributed MOB compaction by procedure > --- > > Key: HBASE-15381 > URL: https://issues.apache.org/jira/browse/HBASE-15381 > Project: HBase > Issue Type: Improvement > Components: mob >Reporter: Jingcheng Du >Assignee: Jingcheng Du >Priority: Major > Attachments: HBASE-15381-v2.patch, HBASE-15381-v3.patch, > HBASE-15381-v4.patch, HBASE-15381-v5.patch, HBASE-15381-v6.patch, > HBASE-15381.patch, mob distributed compaction design-v2.pdf, mob distributed > compaction design.pdf > > > In MOB, there is a periodical compaction which runs in HMaster (It can be > disabled by configuration), some small mob files are merged into bigger ones. > Now the compaction only runs in HMaster which is not efficient and might > impact the running of HMaster. In this JIRA, a distributed MOB compaction is > introduced, it is triggered by HMaster, but all the compaction jobs are > distributed to HRegionServers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-15381) Implement a distributed MOB compaction by procedure
[ https://issues.apache.org/jira/browse/HBASE-15381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun reassigned HBASE-15381: Assignee: huaxiang sun (was: Jingcheng Du) > Implement a distributed MOB compaction by procedure > --- > > Key: HBASE-15381 > URL: https://issues.apache.org/jira/browse/HBASE-15381 > Project: HBase > Issue Type: Improvement > Components: mob >Reporter: Jingcheng Du >Assignee: huaxiang sun >Priority: Major > Attachments: HBASE-15381-v2.patch, HBASE-15381-v3.patch, > HBASE-15381-v4.patch, HBASE-15381-v5.patch, HBASE-15381-v6.patch, > HBASE-15381.patch, mob distributed compaction design-v2.pdf, mob distributed > compaction design.pdf > > > In MOB, there is a periodical compaction which runs in HMaster (It can be > disabled by configuration), some small mob files are merged into bigger ones. > Now the compaction only runs in HMaster which is not efficient and might > impact the running of HMaster. In this JIRA, a distributed MOB compaction is > introduced, it is triggered by HMaster, but all the compaction jobs are > distributed to HRegionServers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19964) TestWriteHeavyIncrementObserver fails
[ https://issues.apache.org/jira/browse/HBASE-19964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19964: -- Resolution: Fixed Status: Resolved (was: Patch Available) +1 Resolving. It dropped off the flakies list. Thanks [~Apache9] > TestWriteHeavyIncrementObserver fails > - > > Key: HBASE-19964 > URL: https://issues.apache.org/jira/browse/HBASE-19964 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19964.patch > > > https://builds.apache.org/job/HBASE-Flaky-Tests/25704/artifact/hbase-examples/target/surefire-reports/org.apache.hadoop.hbase.coprocessor.example.TestWriteHeavyIncrementObserver-output.txt/*view*/ > I could reproduce it locally. Let me dig more. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19863) java.lang.IllegalStateException: isDelete failed when SingleColumnValueFilter is used
[ https://issues.apache.org/jira/browse/HBASE-19863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358803#comment-16358803 ] Josh Elser commented on HBASE-19863: [~ram_krish], [~anoop.hbase], seems to me that you guys understand the problem much better than I do. Any chance I could ask you to keep up on the reviews? I am worried that I don't yet understand the problem well enough to give a proper review alone. > java.lang.IllegalStateException: isDelete failed when SingleColumnValueFilter > is used > - > > Key: HBASE-19863 > URL: https://issues.apache.org/jira/browse/HBASE-19863 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.1 >Reporter: Sergey Soldatov >Assignee: Sergey Soldatov >Priority: Major > Attachments: HBASE-19863-branch1.patch, HBASE-19863-test.patch > > > Under some circumstances scan with SingleColumnValueFilter may fail with an > exception > {noformat} > java.lang.IllegalStateException: isDelete failed: deleteBuffer=C3, > qualifier=C2, timestamp=1516433595543, comparison result: 1 > at > org.apache.hadoop.hbase.regionserver.ScanDeleteTracker.isDeleted(ScanDeleteTracker.java:149) > at > org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.match(ScanQueryMatcher.java:386) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:545) > at > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5876) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6027) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5814) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2552) > at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32385) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167) > {noformat} > Conditions: > table T with a single column family 0 that uses ROWCOL bloom filter > (important) and column qualifiers C1,C2,C3,C4,C5. > When we fill the table for every row we put deleted cell for C3. > The table has a single region with two HStore: > A: start row: 0, stop row: 99 > B: start row: 10 stop row: 99 > B has newer versions of rows 10-99. Store files have several blocks each > (important). > Store A is the result of major compaction, so it doesn't have any deleted > cells (important). > So, we are running a scan like: > {noformat} > scan 'T', { COLUMNS => ['0:C3','0:C5'], FILTER => "SingleColumnValueFilter > ('0','C5',=,'binary:whatever')"} > {noformat} > How the scan performs: > First, we iterate A for rows 0 and 1 without any problems. > Next, we start to iterate A for row 10, so read the first cell and set hfs > scanner to A : > 10:0/C1/0/Put/x but found that we have a newer version of the cell in B : > 10:0/C1/1/Put/x, > so we make B as our current store scanner. Since we are looking for > particular columns > C3 and C5, we perform the optimization StoreScanner.seekOrSkipToNextColumn > which > would run reseek for all store scanners. > For store A the following magic would happen in requestSeek: > 1. bloom filter check passesGeneralBloomFilter would set haveToSeek to > false because row 10 doesn't have C3 qualifier in store A. > 2. Since we don't have to seek we just create a fake row > 10:0/C3/OLDEST_TIMESTAMP/Maximum, an optimization that is quite important for > us and it commented with : > {noformat} > // Multi-column Bloom filter optimization. > // Create a fake key/value, so that this scanner only bubbles up to the > top > // of the KeyValueHeap in StoreScanner after we scanned this row/column in > // all other store files. The query matcher will then just skip this fake > // key/value and the store scanner will progress to the next column. This > // is obviously not a "real real" seek, but unlike the fake KV earlier in > // this method, we want this to be propagated to ScanQueryMatcher. > {noformat} > > For store B we would set it to fake 10:0/C3/createFirstOnRowColTS()/Maximum > to skip C3 entirely. > After that we start searching for qualifier C5 using seekOrSkipToNextColumn > which run first trySkipToNextColumn: > {noformat} > protected boolean trySkipToNe
[jira] [Commented] (HBASE-19876) The exception happening in converting pb mutation to hbase.mutation messes up the CellScanner
[ https://issues.apache.org/jira/browse/HBASE-19876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358840#comment-16358840 ] Hadoop QA commented on HBASE-19876: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m 52s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} branch-1.2 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 23s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 51s{color} | {color:green} branch-1.2 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s{color} | {color:green} branch-1.2 passed with JDK v1.8.0_162 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 11s{color} | {color:green} branch-1.2 passed with JDK v1.7.0_171 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 15s{color} | {color:green} branch-1.2 passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m 18s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 46s{color} | {color:green} branch-1.2 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s{color} | {color:green} branch-1.2 passed with JDK v1.8.0_162 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s{color} | {color:green} branch-1.2 passed with JDK v1.7.0_171 {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s{color} | {color:green} the patch passed with JDK v1.8.0_162 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s{color} | {color:green} the patch passed with JDK v1.7.0_171 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 30s{color} | {color:green} The patch hbase-client passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 15s{color} | {color:green} hbase-server: The patch generated 0 new + 135 unchanged - 1 fixed = 135 total (was 136) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m 27s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 8m 12s{color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1 2.5.2 2.6.5 2.7.4. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s{color} | {color:green} the patch passed with JDK v1.8.0_162 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s{color} | {color:green} the patch passed with JDK v1.7.0_171 {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 53s{color} | {color:green} hbase-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 86m 27s{color} | {color:red} hbase-server in the patch failed. {color} | | {colo
[jira] [Updated] (HBASE-19967) Add Major Compaction Tool options for off-peak / on-peak hours
[ https://issues.apache.org/jira/browse/HBASE-19967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-19967: --- Description: After HBASE-19528 an operator can disable automatic major compaction and more intelligently manage major compaction impact on cluster operations with an external tool that drives the compaction activity. This tool can be invoked at whatever interval is desirable, and can restrict activity by table and column family, with given concurrency of regionservers compacting at a given time. Add Major Compaction Tool options for off-peak / on-peak hours. Allow for definition of "off-peak" as a time range bounded by two points on a 24 hour clock, and for two concurrency target for global compaction activity, one for the off-peak interval, the other for the remainder. was: After HBASE-19528 an operator can disable automatic major compaction and more intelligently manage major compaction impact on cluster operations with an external tool that drives the compaction activity. This tool can be invoked at whatever schedule is desirable, and can restrict activity by table and column family, with given concurrency of regionservers compacting at a given time. Add Major Compaction Tool options for off-peak / on-peak hours. Allow for definition of "off-peak" as a time range bounded by two points on a 24 hour clock, and for two concurrency target for global compaction activity, one for the off-peak interval, the other for the remainder. > Add Major Compaction Tool options for off-peak / on-peak hours > -- > > Key: HBASE-19967 > URL: https://issues.apache.org/jira/browse/HBASE-19967 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Priority: Major > > After HBASE-19528 an operator can disable automatic major compaction and more > intelligently manage major compaction impact on cluster operations with an > external tool that drives the compaction activity. This tool can be invoked > at whatever interval is desirable, and can restrict activity by table and > column family, with given concurrency of regionservers compacting at a given > time. > Add Major Compaction Tool options for off-peak / on-peak hours. Allow for > definition of "off-peak" as a time range bounded by two points on a 24 hour > clock, and for two concurrency target for global compaction activity, one for > the off-peak interval, the other for the remainder. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-19967) Add Major Compaction Tool options for off-peak / on-peak hours
Andrew Purtell created HBASE-19967: -- Summary: Add Major Compaction Tool options for off-peak / on-peak hours Key: HBASE-19967 URL: https://issues.apache.org/jira/browse/HBASE-19967 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell After HBASE-19528 an operator can disable automatic major compaction and more intelligently manage major compaction impact on cluster operations with an external tool that drives the compaction activity. This tool can be invoked at whatever schedule is desirable, and can restrict activity by table and column family, with given concurrency of regionservers compacting at a given time. Add Major Compaction Tool options for off-peak / on-peak hours. Allow for definition of "off-peak" as a time range bounded by two points on a 24 hour clock, and for two concurrency target for global compaction activity, one for the off-peak interval, the other for the remainder. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[ https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359031#comment-16359031 ] Vladimir Rodionov commented on HBASE-17825: --- {quote} When false is returned, some files would be deleted while some are left. Is this desirable? {quote} This needs to be added to FT Jira. We need to restore not only meta in case of failure but physical data as well. > Backup: further optimizations > - > > Key: HBASE-17825 > URL: https://issues.apache.org/jira/browse/HBASE-17825 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 3.0.0 > > Attachments: HBASE-17825-v1.patch > > > Some phases of backup and restore can be optimized: > # WALPlayer support for multiple tables > # Run DistCp once per all tables during backup/restore > The eventual goal: > # 2 M/R jobs per backup/restore -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-19968) MapReduce test fails with NoClassDefFoundError against hadoop3
Ted Yu created HBASE-19968: -- Summary: MapReduce test fails with NoClassDefFoundError against hadoop3 Key: HBASE-19968 URL: https://issues.apache.org/jira/browse/HBASE-19968 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu When running mapreduce tests against hadoop3, I observed the following: {code} [ERROR] testWithMockedMapReduceSingleRegion(org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat) Time elapsed: 0.024 s <<< ERROR! java.lang.NoClassDefFoundError: org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo at org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat.testWithMockedMapReduce(TestTableSnapshotInputFormat.java:178) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo at org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat.testWithMockedMapReduce(TestTableSnapshotInputFormat.java:178) {code} This was due to lack of dependency on hadoop-hdfs module in the hadoop-3.0 profile of hbase-mapreduce/pom.xml -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19968) MapReduce test fails with NoClassDefFoundError against hadoop3
[ https://issues.apache.org/jira/browse/HBASE-19968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359043#comment-16359043 ] Ted Yu commented on HBASE-19968: TestTableSnapshotInputFormat passed with the patch against hadoop3. Running mapreduce tests against hadoop3 now. > MapReduce test fails with NoClassDefFoundError against hadoop3 > -- > > Key: HBASE-19968 > URL: https://issues.apache.org/jira/browse/HBASE-19968 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 19968.v1.txt > > > When running mapreduce tests against hadoop3, I observed the following: > {code} > [ERROR] > testWithMockedMapReduceSingleRegion(org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat) > Time elapsed: 0.024 s <<< ERROR! > java.lang.NoClassDefFoundError: > org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo > at > org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat.testWithMockedMapReduce(TestTableSnapshotInputFormat.java:178) > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo > at > org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat.testWithMockedMapReduce(TestTableSnapshotInputFormat.java:178) > {code} > This was due to lack of dependency on hadoop-hdfs module in the hadoop-3.0 > profile of hbase-mapreduce/pom.xml -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19968) MapReduce test fails with NoClassDefFoundError against hadoop3
[ https://issues.apache.org/jira/browse/HBASE-19968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-19968: --- Attachment: 19968.v1.txt > MapReduce test fails with NoClassDefFoundError against hadoop3 > -- > > Key: HBASE-19968 > URL: https://issues.apache.org/jira/browse/HBASE-19968 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 19968.v1.txt > > > When running mapreduce tests against hadoop3, I observed the following: > {code} > [ERROR] > testWithMockedMapReduceSingleRegion(org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat) > Time elapsed: 0.024 s <<< ERROR! > java.lang.NoClassDefFoundError: > org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo > at > org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat.testWithMockedMapReduce(TestTableSnapshotInputFormat.java:178) > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo > at > org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat.testWithMockedMapReduce(TestTableSnapshotInputFormat.java:178) > {code} > This was due to lack of dependency on hadoop-hdfs module in the hadoop-3.0 > profile of hbase-mapreduce/pom.xml -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19968) MapReduce test fails with NoClassDefFoundError against hadoop3
[ https://issues.apache.org/jira/browse/HBASE-19968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-19968: --- Status: Patch Available (was: Open) > MapReduce test fails with NoClassDefFoundError against hadoop3 > -- > > Key: HBASE-19968 > URL: https://issues.apache.org/jira/browse/HBASE-19968 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 19968.v1.txt > > > When running mapreduce tests against hadoop3, I observed the following: > {code} > [ERROR] > testWithMockedMapReduceSingleRegion(org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat) > Time elapsed: 0.024 s <<< ERROR! > java.lang.NoClassDefFoundError: > org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo > at > org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat.testWithMockedMapReduce(TestTableSnapshotInputFormat.java:178) > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo > at > org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat.testWithMockedMapReduce(TestTableSnapshotInputFormat.java:178) > {code} > This was due to lack of dependency on hadoop-hdfs module in the hadoop-3.0 > profile of hbase-mapreduce/pom.xml -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19949) TestRSGroupsWithACL fails with ExceptionInInitializerError
[ https://issues.apache.org/jira/browse/HBASE-19949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-19949: --- Fix Version/s: 1.4.2 1.5.0 > TestRSGroupsWithACL fails with ExceptionInInitializerError > -- > > Key: HBASE-19949 > URL: https://issues.apache.org/jira/browse/HBASE-19949 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: 1.5.0, 2.0.0-beta-2, 1.4.2 > > Attachments: 19949.v1.txt > > > {code} > org.apache.hadoop.hbase.rsgroup.TestRSGroupsWithACL Time elapsed: 0.168 sec > <<< ERROR! > java.lang.ExceptionInInitializerError > at > org.apache.hadoop.hbase.rsgroup.TestRSGroupsWithACL.(TestRSGroupsWithACL.java:60) > {code} > The cause is that HBaseClassTestRule#getTimeoutInSeconds() cannot retrieve > annotation for test size. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-16060) 1.x clients cannot access table state talking to 2.0 cluster
[ https://issues.apache.org/jira/browse/HBASE-16060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359069#comment-16359069 ] stack commented on HBASE-16060: --- With patch in place... hbase(main):002:0> is_enabled 'TestTable' true 0 row(s) in 0.1240 seconds hbase(main):003:0> is_enabled 'disable' false 0 row(s) in 0.0230 seconds hbase(main):004:0> is_disabled 'disable' true 0 row(s) in 0.0190 seconds hbase(main):005:0> is_disabled 'TestTable' false 0 row(s) in 0.0250 seconds 'disable' is a disabled table. Playing more. The migration of table state from hbase-1.x. zookeepers is something we want and we have something courtesy of HBASE-13032 but it deleted all zk table state when done which messes up what this patch is trying to do. > 1.x clients cannot access table state talking to 2.0 cluster > > > Key: HBASE-16060 > URL: https://issues.apache.org/jira/browse/HBASE-16060 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: stack >Priority: Blocker > Fix For: 2.0.0-beta-2 > > Attachments: > 0002-HBASE-16060-1.x-clients-cannot-access-table-state-ta.patch, > HBASE-16060.branch-2.001.patch > > > Since table state is migrated to meta instead of zk in 2.0, 1.x clients > talking to 2.0 cluster cannot access the table state. This causes some weird > behavior since from a client perspective, {{Admin.isTableEnabled()}} and > {{Admin.isTableDisabled()}} both return false. > One option we can do is to add code in 1.x clients so that they can access > the table state in meta if needed. Otherwise, we can mirror the table state > in zk (while keeping meta as the source of truth) during 2.x lifecycle so > that any 1.x client can still work correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19949) TestRSGroupsWithACL fails with ExceptionInInitializerError
[ https://issues.apache.org/jira/browse/HBASE-19949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-19949: --- Fix Version/s: (was: 1.4.2) (was: 1.5.0) > TestRSGroupsWithACL fails with ExceptionInInitializerError > -- > > Key: HBASE-19949 > URL: https://issues.apache.org/jira/browse/HBASE-19949 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: 19949.v1.txt > > > {code} > org.apache.hadoop.hbase.rsgroup.TestRSGroupsWithACL Time elapsed: 0.168 sec > <<< ERROR! > java.lang.ExceptionInInitializerError > at > org.apache.hadoop.hbase.rsgroup.TestRSGroupsWithACL.(TestRSGroupsWithACL.java:60) > {code} > The cause is that HBaseClassTestRule#getTimeoutInSeconds() cannot retrieve > annotation for test size. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-19969) Improve FT in merge operation
Vladimir Rodionov created HBASE-19969: - Summary: Improve FT in merge operation Key: HBASE-19969 URL: https://issues.apache.org/jira/browse/HBASE-19969 Project: HBase Issue Type: Sub-task Reporter: Vladimir Rodionov -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-19969) Improve FT in merge operation
[ https://issues.apache.org/jira/browse/HBASE-19969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov reassigned HBASE-19969: - Assignee: Vladimir Rodionov > Improve FT in merge operation > - > > Key: HBASE-19969 > URL: https://issues.apache.org/jira/browse/HBASE-19969 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19859) Update download page header for 1.1 EOL
[ https://issues.apache.org/jira/browse/HBASE-19859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359074#comment-16359074 ] Andrew Purtell commented on HBASE-19859: After we move the stable pointer to 1.3.2 I'd like to work towards moving again to 1.4. I'll start a discussion on dev@ at that time to establish acceptance criteria. > Update download page header for 1.1 EOL > --- > > Key: HBASE-19859 > URL: https://issues.apache.org/jira/browse/HBASE-19859 > Project: HBase > Issue Type: Task >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Attachments: HBASE-19583.patch > > > See example mirror: http://mirrors.ocf.berkeley.edu/apache/hbase/ > They still claim that 1.1 is under active development. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19969) Improve FT in merge operation
[ https://issues.apache.org/jira/browse/HBASE-19969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-19969: -- Description: Some file system operations are not fault tolerant during merge. We delete backup data in a backup file system, then copy new data over to backup destination. Deletes can be partial, copy can fail as well > Improve FT in merge operation > - > > Key: HBASE-19969 > URL: https://issues.apache.org/jira/browse/HBASE-19969 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Major > > Some file system operations are not fault tolerant during merge. We delete > backup data in a backup file system, then copy new data over to backup > destination. Deletes can be partial, copy can fail as well -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-17825) Backup: further optimizations
[ https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359031#comment-16359031 ] Vladimir Rodionov edited comment on HBASE-17825 at 2/9/18 10:59 PM: {quote} When false is returned, some files would be deleted while some are left. Is this desirable? {quote} This needs to be added to FT Jira. We need to restore not only meta in case of failure but physical data as well. That is the separate issue: HBASE-19969 was (Author: vrodionov): {quote} When false is returned, some files would be deleted while some are left. Is this desirable? {quote} This needs to be added to FT Jira. We need to restore not only meta in case of failure but physical data as well. > Backup: further optimizations > - > > Key: HBASE-17825 > URL: https://issues.apache.org/jira/browse/HBASE-17825 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 3.0.0 > > Attachments: HBASE-17825-v1.patch > > > Some phases of backup and restore can be optimized: > # WALPlayer support for multiple tables > # Run DistCp once per all tables during backup/restore > The eventual goal: > # 2 M/R jobs per backup/restore -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19937) Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer are initialized
[ https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-19937: --- Fix Version/s: 1.4.2 1.5.0 > Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer > are initialized > - > > Key: HBASE-19937 > URL: https://issues.apache.org/jira/browse/HBASE-19937 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.0.0-beta-2 >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > Fix For: 1.5.0, 2.0.0-beta-2, 1.4.2 > > Attachments: HBASE-19937.branch-2.001.patch, > HBASE-19937.branch-2.002.patch, HBASE-19937.branch-2.003.patch, > HBASE-19937.branch-2.004.patch, HBASE-19937.branch-2.005.patch, > HBASE-19937.branch-2.006.patch, import-order.png > > > hbase:rsgroup table will be created by calling createRSGroupTable when master > load system coprocessors in > > {code:java} > 844 this.cpHost = new MasterCoprocessorHost(this, this.conf);{code} > when ProcedureExecutor hasn't been initialized before createRSGroupTable, it > will encounter Exception as follows, > > {noformat} > Exception in thread > "org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130" > java.lang.IllegalArgumentException > at > org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835) > at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795) > at > org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858) > at > org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823) > at > org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat} > while ProcedureExecutor initialized by calling > {code:java} > 848 startServiceThreads();{code} > And LoadBalancer is initialized by calling > {code:java} > 868 this.balancer.initialize();{code} > When LoadBalancer hasn't been initialized before createRSGroupTable, it will > encounters Exception as follows, > 2018-02-02,16:12:45,688 ERROR > org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught > runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; > CreateTableProcedure table=hbase:rsgroup > java.lang.NullPointerException > at > org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254) > at > org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162) > at > org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603) > at > org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108) > at > org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51) > at > org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182) > at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738) > > As a result of CreateTableProcedure.rollbackState, it may then print logs > warning TableExistsException as follows, > 2018-02-02,16:12:55,503 WARN > org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: > Failed to perform check > java.io.IOException: Failed to create group table. > org.apache.hadoop.hbase.TableExistsException: hbase:rsgroup > at > org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877) > > After some auto-retries, it loops running the thread RSGroupStartupWorker, > will print logs as follows, > 2018-02-02,16:23:17,626 INFO > org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: > RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0,
[jira] [Updated] (HBASE-17825) Backup: further optimizations
[ https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-17825: -- Attachment: HBASE-17825-v2.patch > Backup: further optimizations > - > > Key: HBASE-17825 > URL: https://issues.apache.org/jira/browse/HBASE-17825 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 3.0.0 > > Attachments: HBASE-17825-v1.patch, HBASE-17825-v2.patch > > > Some phases of backup and restore can be optimized: > # WALPlayer support for multiple tables > # Run DistCp once per all tables during backup/restore > The eventual goal: > # 2 M/R jobs per backup/restore -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17825) Backup: further optimizations
[ https://issues.apache.org/jira/browse/HBASE-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359076#comment-16359076 ] Vladimir Rodionov commented on HBASE-17825: --- Patch v2 contains minor fixes. Created HBASE-19969 to improve FT during Merge > Backup: further optimizations > - > > Key: HBASE-17825 > URL: https://issues.apache.org/jira/browse/HBASE-17825 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 3.0.0 > > Attachments: HBASE-17825-v1.patch, HBASE-17825-v2.patch > > > Some phases of backup and restore can be optimized: > # WALPlayer support for multiple tables > # Run DistCp once per all tables during backup/restore > The eventual goal: > # 2 M/R jobs per backup/restore -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19968) MapReduce test fails with NoClassDefFoundError against hadoop3
[ https://issues.apache.org/jira/browse/HBASE-19968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359079#comment-16359079 ] Hadoop QA commented on HBASE-19968: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s{color} | {color:blue} Docker mode activated. {color} | | {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 1s{color} | {color:blue} The patch file was not named according to hbase's naming conventions. Please see https://yetus.apache.org/documentation/0.7.0/precommit-patchnames for instructions. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 21s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 9m 28s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 1s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 41s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 18m 16s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m 7s{color} | {color:green} hbase-mapreduce in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 11s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 45m 39s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 | | JIRA Issue | HBASE-19968 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12909990/19968.v1.txt | | Optional Tests | asflicense javac javadoc unit shadedjars hadoopcheck xml compile | | uname | Linux babcae924bfd 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 2b63af376e | | maven | version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) | | Default Java | 1.8.0_151 | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/11469/testReport/ | | Max. process+thread count | 4179 (vs. ulimit of 1) | | modules | C: hbase-mapreduce U: hbase-mapreduce | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/11469/console | | Powered by | Apache Yetus 0.7.0 http://yetus.apache.org | This message was automatically generated. > MapReduce test fails with NoClassDefFoundError against hadoop3 > -- > > Key: HBASE-19968 > URL: https://issues.apa
[jira] [Updated] (HBASE-19949) TestRSGroupsWithACL fails with ExceptionInInitializerError
[ https://issues.apache.org/jira/browse/HBASE-19949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-19949: --- Fix Version/s: 1.4.2 1.5.0 > TestRSGroupsWithACL fails with ExceptionInInitializerError > -- > > Key: HBASE-19949 > URL: https://issues.apache.org/jira/browse/HBASE-19949 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: 1.5.0, 2.0.0-beta-2, 1.4.2 > > Attachments: 19949.v1.txt > > > {code} > org.apache.hadoop.hbase.rsgroup.TestRSGroupsWithACL Time elapsed: 0.168 sec > <<< ERROR! > java.lang.ExceptionInInitializerError > at > org.apache.hadoop.hbase.rsgroup.TestRSGroupsWithACL.(TestRSGroupsWithACL.java:60) > {code} > The cause is that HBaseClassTestRule#getTimeoutInSeconds() cannot retrieve > annotation for test size. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-16060) 1.x clients cannot access table state talking to 2.0 cluster
[ https://issues.apache.org/jira/browse/HBASE-16060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359099#comment-16359099 ] stack commented on HBASE-16060: --- Ok. First set up cluster with hbase-1 data. Shut it down. Start hbase-2 over hbase-1 data. In logs I see these messages: 2018-02-09 15:28:40,980 INFO [master/ve0524:16000] master.MirroringTableStateManager: Migrating table state from zookeeper to hbase:meta; tableName=enabled2, state=ENABLED 2018-02-09 15:28:41,166 INFO [master/ve0524:16000] hbase.MetaTableAccessor: Updated table enabled2 state to ENABLED in META 2018-02-09 15:28:41,177 INFO [master/ve0524:16000] master.MirroringTableStateManager: Migrating table state from zookeeper to hbase:meta; tableName=hbase:namespace, state=ENABLED 2018-02-09 15:28:41,189 INFO [master/ve0524:16000] hbase.MetaTableAccessor: Updated table hbase:namespace state to ENABLED in META 2018-02-09 15:28:41,200 INFO [master/ve0524:16000] master.MirroringTableStateManager: Purged table state entry for table not in hbase:meta: disable 2018-02-09 15:28:41,210 INFO [master/ve0524:16000] master.MirroringTableStateManager: Migrating table state from zookeeper to hbase:meta; tableName=disabled, state=DISABLED ... so migration of table state from zk to hbase:meta seems to work. Restart and no messages (though we are doing the work to see if we need to migrate states... ). > 1.x clients cannot access table state talking to 2.0 cluster > > > Key: HBASE-16060 > URL: https://issues.apache.org/jira/browse/HBASE-16060 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: stack >Priority: Blocker > Fix For: 2.0.0-beta-2 > > Attachments: > 0002-HBASE-16060-1.x-clients-cannot-access-table-state-ta.patch, > HBASE-16060.branch-2.001.patch > > > Since table state is migrated to meta instead of zk in 2.0, 1.x clients > talking to 2.0 cluster cannot access the table state. This causes some weird > behavior since from a client perspective, {{Admin.isTableEnabled()}} and > {{Admin.isTableDisabled()}} both return false. > One option we can do is to add code in 1.x clients so that they can access > the table state in meta if needed. Otherwise, we can mirror the table state > in zk (while keeping meta as the source of truth) during 2.x lifecycle so > that any 1.x client can still work correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-16060) 1.x clients cannot access table state talking to 2.0 cluster
[ https://issues.apache.org/jira/browse/HBASE-16060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359106#comment-16359106 ] stack commented on HBASE-16060: --- Hmm.. With the patch I can now create tables using a 1.x client. I can get detailed status too. Seems like most function is now working (I've not done comprehensive test). > 1.x clients cannot access table state talking to 2.0 cluster > > > Key: HBASE-16060 > URL: https://issues.apache.org/jira/browse/HBASE-16060 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: stack >Priority: Blocker > Fix For: 2.0.0-beta-2 > > Attachments: > 0002-HBASE-16060-1.x-clients-cannot-access-table-state-ta.patch, > HBASE-16060.branch-2.001.patch > > > Since table state is migrated to meta instead of zk in 2.0, 1.x clients > talking to 2.0 cluster cannot access the table state. This causes some weird > behavior since from a client perspective, {{Admin.isTableEnabled()}} and > {{Admin.isTableDisabled()}} both return false. > One option we can do is to add code in 1.x clients so that they can access > the table state in meta if needed. Otherwise, we can mirror the table state > in zk (while keeping meta as the source of truth) during 2.x lifecycle so > that any 1.x client can still work correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-16060) 1.x clients cannot access table state talking to 2.0 cluster
[ https://issues.apache.org/jira/browse/HBASE-16060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16060: -- Attachment: HBASE-16060.branch-2.002.patch > 1.x clients cannot access table state talking to 2.0 cluster > > > Key: HBASE-16060 > URL: https://issues.apache.org/jira/browse/HBASE-16060 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: stack >Priority: Blocker > Fix For: 2.0.0-beta-2 > > Attachments: > 0002-HBASE-16060-1.x-clients-cannot-access-table-state-ta.patch, > HBASE-16060.branch-2.001.patch, HBASE-16060.branch-2.002.patch > > > Since table state is migrated to meta instead of zk in 2.0, 1.x clients > talking to 2.0 cluster cannot access the table state. This causes some weird > behavior since from a client perspective, {{Admin.isTableEnabled()}} and > {{Admin.isTableDisabled()}} both return false. > One option we can do is to add code in 1.x clients so that they can access > the table state in meta if needed. Otherwise, we can mirror the table state > in zk (while keeping meta as the source of truth) during 2.x lifecycle so > that any 1.x client can still work correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-16060) 1.x clients cannot access table state talking to 2.0 cluster
[ https://issues.apache.org/jira/browse/HBASE-16060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16060: -- Release Note: By default, we mirror table state to zookeeper so hbase-1.x clients will work against an hbase-2 cluster (With this patch, hbase-1.x clients can do most Admin functions including table create; hbase-1.x clients can do all Table/DML against hbase-2 cluster). Flag to disable mirroring is hbase.mirror.table.state.to.zookeeper; set it to false in Configuration. Related, Master on startup will look to see if there are table state znodes left over by an hbase-1 instance. If any found, it will migrate the table state to hbase-2 setting the state into the hbase:meta table where table state is now kept. We will do this check on every Master start. Notion is that this will be overall beneficial with low impediment. To disable the migration check, set hbase.migrate.table.state.from.zookeeper to false. Status: Patch Available (was: Open) > 1.x clients cannot access table state talking to 2.0 cluster > > > Key: HBASE-16060 > URL: https://issues.apache.org/jira/browse/HBASE-16060 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: stack >Priority: Blocker > Fix For: 2.0.0-beta-2 > > Attachments: > 0002-HBASE-16060-1.x-clients-cannot-access-table-state-ta.patch, > HBASE-16060.branch-2.001.patch, HBASE-16060.branch-2.002.patch > > > Since table state is migrated to meta instead of zk in 2.0, 1.x clients > talking to 2.0 cluster cannot access the table state. This causes some weird > behavior since from a client perspective, {{Admin.isTableEnabled()}} and > {{Admin.isTableDisabled()}} both return false. > One option we can do is to add code in 1.x clients so that they can access > the table state in meta if needed. Otherwise, we can mirror the table state > in zk (while keeping meta as the source of truth) during 2.x lifecycle so > that any 1.x client can still work correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-19172) [rolling upgrade] Migrate region state from zk to hbase:meta
[ https://issues.apache.org/jira/browse/HBASE-19172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-19172. --- Resolution: Duplicate HBASE-13032 did a form of this years ago. HBASE-13032 was refactored in HBASE-16060 which adds a unit test to prove this works. It also includes notes on how we verified this working on a cluster deploy. Resolving as duplicate. > [rolling upgrade] Migrate region state from zk to hbase:meta > > > Key: HBASE-19172 > URL: https://issues.apache.org/jira/browse/HBASE-19172 > Project: HBase > Issue Type: Sub-task > Components: migration >Reporter: stack >Priority: Major > Fix For: 2.0.0-beta-2 > > > HBASE-19165 is about starting a branch-2 over branch-1 dataset. The Master > when it comes up, goes to read the 'state' of the region from hbase:meta > because this is how region state is kept now by AMv2 only, there is no state > in hbase:meta on a fresh start. In hbase1, region state was kept in master > memory persisted to zk (rather than hbase:meta) while the region was in > transition. > We need to add to master a self-migration facility. If a region has no state > in hbase:meta, then look in zk to see if it RIT. If nothing in RIT, ask the > pertinent regionserver if it has the region. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-16060) 1.x clients cannot access table state talking to 2.0 cluster
[ https://issues.apache.org/jira/browse/HBASE-16060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359123#comment-16359123 ] stack commented on HBASE-16060: --- .001 does the below. May I have a review please? HBASE-16060 1.x clients cannot access table state talking to 2.0 cluster This patch adds mirroring of table state out to zookeeper. HBase-1.x clients look for table state in zookeeper, not in hbase:meta where hbase-2.x maintains table state. The patch also moves and refactors the 'migration' code that was put in place by HBASE-13032. D hbase-client/src/main/java/org/apache/hadoop/hbase/CoordinatedStateException.java Unused. M hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java Move table state migration code from Master startup out to TableStateManager where it belongs. Also start MirroringTableStateManager dependent on config. A hbase-server/src/main/java/org/apache/hadoop/hbase/master/MirroringTableStateManager.java M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java Move migration from zookeeper of table state in here. Also plumb in mechanism so subclass can get a chance to look at table state as we do the startup fixup full-table scan of meta. M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java Bug-fix. Now we create regions in CLOSED state but we fail to check table state; were presuming table always enabled. Meant on startup there'd be an unassigned region that never got assigned. A hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMirroringTableStateManager.java Test migration and mirroring. > 1.x clients cannot access table state talking to 2.0 cluster > > > Key: HBASE-16060 > URL: https://issues.apache.org/jira/browse/HBASE-16060 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: stack >Priority: Blocker > Fix For: 2.0.0-beta-2 > > Attachments: > 0002-HBASE-16060-1.x-clients-cannot-access-table-state-ta.patch, > HBASE-16060.branch-2.001.patch, HBASE-16060.branch-2.002.patch > > > Since table state is migrated to meta instead of zk in 2.0, 1.x clients > talking to 2.0 cluster cannot access the table state. This causes some weird > behavior since from a client perspective, {{Admin.isTableEnabled()}} and > {{Admin.isTableDisabled()}} both return false. > One option we can do is to add code in 1.x clients so that they can access > the table state in meta if needed. Otherwise, we can mirror the table state > in zk (while keeping meta as the source of truth) during 2.x lifecycle so > that any 1.x client can still work correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19942) Fix flaky TestSimpleRpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-19942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19942: -- Resolution: Fixed Status: Resolved (was: Patch Available) Re-resolving after [~zghaobac] 's push. This test no longer shows on the flakies list. Thank you [~zghaobac] > Fix flaky TestSimpleRpcScheduler > > > Key: HBASE-19942 > URL: https://issues.apache.org/jira/browse/HBASE-19942 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19942.branch-2.001.patch, > HBASE-19942.branch-2.addendum.patch, HBASE-19942.master.001.patch, > HBASE-19942.master.addendum.patch > > > [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests-branch2.0/lastSuccessfulBuild/artifact/dashboard.html] > > https://builds.apache.org/job/HBASE-Flaky-Tests-branch2.0/1387/testReport/junit/org.apache.hadoop.hbase.ipc/TestSimpleRpcScheduler/testSoftAndHardQueueLimits/ > > h3. Stacktrace > java.lang.AssertionError at > org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler.testSoftAndHardQueueLimits(TestSimpleRpcScheduler.java:451) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19966) The WriteEntry for WALKey maybe null if we failed to call WAL.append
[ https://issues.apache.org/jira/browse/HBASE-19966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19966: -- Attachment: 0001-HBASE-16060-1.x-clients-cannot-access-table-state-ta.patch > The WriteEntry for WALKey maybe null if we failed to call WAL.append > > > Key: HBASE-19966 > URL: https://issues.apache.org/jira/browse/HBASE-19966 > Project: HBase > Issue Type: Bug > Components: wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19966.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19966) The WriteEntry for WALKey maybe null if we failed to call WAL.append
[ https://issues.apache.org/jira/browse/HBASE-19966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19966: -- Attachment: (was: 0001-HBASE-16060-1.x-clients-cannot-access-table-state-ta.patch) > The WriteEntry for WALKey maybe null if we failed to call WAL.append > > > Key: HBASE-19966 > URL: https://issues.apache.org/jira/browse/HBASE-19966 > Project: HBase > Issue Type: Bug > Components: wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19966.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19966) The WriteEntry for WALKey maybe null if we failed to call WAL.append
[ https://issues.apache.org/jira/browse/HBASE-19966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19966: -- Attachment: HBASE-19966.patch > The WriteEntry for WALKey maybe null if we failed to call WAL.append > > > Key: HBASE-19966 > URL: https://issues.apache.org/jira/browse/HBASE-19966 > Project: HBase > Issue Type: Bug > Components: wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19966.patch, HBASE-19966.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19966) The WriteEntry for WALKey maybe null if we failed to call WAL.append
[ https://issues.apache.org/jira/browse/HBASE-19966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359132#comment-16359132 ] stack commented on HBASE-19966: --- Ignore above comment. Not going to happen. Meantime, retrying while Duo having his breakfast. > The WriteEntry for WALKey maybe null if we failed to call WAL.append > > > Key: HBASE-19966 > URL: https://issues.apache.org/jira/browse/HBASE-19966 > Project: HBase > Issue Type: Bug > Components: wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19966.patch, HBASE-19966.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-16060) 1.x clients cannot access table state talking to 2.0 cluster
[ https://issues.apache.org/jira/browse/HBASE-16060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359205#comment-16359205 ] Hadoop QA commented on HBASE-16060: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 1s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} branch-2 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 21s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 30s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 45s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m 45s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s{color} | {color:green} branch-2 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 9s{color} | {color:green} The patch hbase-protocol-shaded passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 29s{color} | {color:green} hbase-client: The patch generated 0 new + 26 unchanged - 1 fixed = 26 total (was 27) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 8s{color} | {color:red} hbase-server: The patch generated 16 new + 294 unchanged - 5 fixed = 310 total (was 299) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 2s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 14m 47s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 28s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 53s{color} | {color:green} hbase-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 41s{color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 56s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}140m 23s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop
[jira] [Commented] (HBASE-19966) The WriteEntry for WALKey maybe null if we failed to call WAL.append
[ https://issues.apache.org/jira/browse/HBASE-19966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359206#comment-16359206 ] Hadoop QA commented on HBASE-19966: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 56s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 55s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m 10s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 56s{color} | {color:red} hbase-server: The patch generated 3 new + 13 unchanged - 6 fixed = 16 total (was 19) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 14s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 15s{color} | {color:red} The patch causes 10 errors with Hadoop v2.6.5. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 13s{color} | {color:red} The patch causes 10 errors with Hadoop v2.7.4. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m 23s{color} | {color:red} The patch causes 10 errors with Hadoop v3.0.0. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 98m 55s{color} | {color:green} hbase-server in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 22s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}126m 20s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 | | JIRA Issue | HBASE-19966 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12910005/HBASE-19966.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 05e8a8f1b0fa 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 2b63af376e | | maven | version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) | | Default Java | 1.8.0_151 | | checkstyle | https://builds.apache.org/job/PreCommit-HBASE-Build/11471/artifact/patchprocess/diff-checkstyle-hbase-server.txt | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/114
[jira] [Created] (HBASE-19970) Remove unused functions from TableAuthManager
Appy created HBASE-19970: Summary: Remove unused functions from TableAuthManager Key: HBASE-19970 URL: https://issues.apache.org/jira/browse/HBASE-19970 Project: HBase Issue Type: Task Reporter: Appy Assignee: Appy Functions deleted in TableAuthManager: - setTableUserPermissions - setTableGroupPermissions - setNamespaceUserPermissions - setNamespaceGroupPermissions - writeTableToZooKeeper - writeNamespaceToZooKeeper To make sure it was not a bug, and that relevant functionality moved to some alternate code path, tried to find out why and when these functions went out of use. But just couldn't figure out...until i reached the patch which added them. Looks like they were dead functions to start with :) Jira which added them: HBASE-8409. Commit id: ac10b3c13d6b66e12d0c9601204b01dfa525ed19 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19970) Remove unused functions from TableAuthManager
[ https://issues.apache.org/jira/browse/HBASE-19970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359209#comment-16359209 ] Appy commented on HBASE-19970: -- I was trying to understand read/write paths for ACL meta data, looks like the write path is: grant() rpc --> execute CP service on ACL table's region --> write data to ACL region --> postGet hook --> use ZKPermissionsWatcher to write new data to zk > Remove unused functions from TableAuthManager > - > > Key: HBASE-19970 > URL: https://issues.apache.org/jira/browse/HBASE-19970 > Project: HBase > Issue Type: Task >Reporter: Appy >Assignee: Appy >Priority: Minor > > Functions deleted in TableAuthManager: > - setTableUserPermissions > - setTableGroupPermissions > - setNamespaceUserPermissions > - setNamespaceGroupPermissions > - writeTableToZooKeeper > - writeNamespaceToZooKeeper > To make sure it was not a bug, and that relevant functionality moved to some > alternate code path, tried to find out why and when these functions went out > of use. But just couldn't figure out...until i reached the patch which added > them. Looks like they were dead functions to start with :) > Jira which added them: HBASE-8409. Commit id: > ac10b3c13d6b66e12d0c9601204b01dfa525ed19 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-19970) Remove unused functions from TableAuthManager
[ https://issues.apache.org/jira/browse/HBASE-19970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359209#comment-16359209 ] Appy edited comment on HBASE-19970 at 2/10/18 2:37 AM: --- I was trying to understand read/write paths for ACL meta data, looks like the write path is as following (and doesn't involve deleted functions in any way): grant() rpc --> execute CP service on ACL table's region --> write data to ACL region --> postGet hook --> use ZKPermissionsWatcher to write new data to zk was (Author: appy): I was trying to understand read/write paths for ACL meta data, looks like the write path is: grant() rpc --> execute CP service on ACL table's region --> write data to ACL region --> postGet hook --> use ZKPermissionsWatcher to write new data to zk > Remove unused functions from TableAuthManager > - > > Key: HBASE-19970 > URL: https://issues.apache.org/jira/browse/HBASE-19970 > Project: HBase > Issue Type: Task >Reporter: Appy >Assignee: Appy >Priority: Minor > > Functions deleted in TableAuthManager: > - setTableUserPermissions > - setTableGroupPermissions > - setNamespaceUserPermissions > - setNamespaceGroupPermissions > - writeTableToZooKeeper > - writeNamespaceToZooKeeper > To make sure it was not a bug, and that relevant functionality moved to some > alternate code path, tried to find out why and when these functions went out > of use. But just couldn't figure out...until i reached the patch which added > them. Looks like they were dead functions to start with :) > Jira which added them: HBASE-8409. Commit id: > ac10b3c13d6b66e12d0c9601204b01dfa525ed19 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19970) Remove unused functions from TableAuthManager
[ https://issues.apache.org/jira/browse/HBASE-19970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-19970: - Attachment: HBASE-19970.master.001.patch > Remove unused functions from TableAuthManager > - > > Key: HBASE-19970 > URL: https://issues.apache.org/jira/browse/HBASE-19970 > Project: HBase > Issue Type: Task >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-19970.master.001.patch > > > Functions deleted in TableAuthManager: > - setTableUserPermissions > - setTableGroupPermissions > - setNamespaceUserPermissions > - setNamespaceGroupPermissions > - writeTableToZooKeeper > - writeNamespaceToZooKeeper > To make sure it was not a bug, and that relevant functionality moved to some > alternate code path, tried to find out why and when these functions went out > of use. But just couldn't figure out...until i reached the patch which added > them. Looks like they were dead functions to start with :) > Jira which added them: HBASE-8409. Commit id: > ac10b3c13d6b66e12d0c9601204b01dfa525ed19 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19970) Remove unused functions from TableAuthManager
[ https://issues.apache.org/jira/browse/HBASE-19970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359215#comment-16359215 ] Appy commented on HBASE-19970: -- Ping [~stack] > Remove unused functions from TableAuthManager > - > > Key: HBASE-19970 > URL: https://issues.apache.org/jira/browse/HBASE-19970 > Project: HBase > Issue Type: Task >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-19970.master.001.patch > > > Functions deleted in TableAuthManager: > - setTableUserPermissions > - setTableGroupPermissions > - setNamespaceUserPermissions > - setNamespaceGroupPermissions > - writeTableToZooKeeper > - writeNamespaceToZooKeeper > To make sure it was not a bug, and that relevant functionality moved to some > alternate code path, tried to find out why and when these functions went out > of use. But just couldn't figure out...until i reached the patch which added > them. Looks like they were dead functions to start with :) > Jira which added them: HBASE-8409. Commit id: > ac10b3c13d6b66e12d0c9601204b01dfa525ed19 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19970) Remove unused functions from TableAuthManager
[ https://issues.apache.org/jira/browse/HBASE-19970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-19970: - Status: Patch Available (was: Open) > Remove unused functions from TableAuthManager > - > > Key: HBASE-19970 > URL: https://issues.apache.org/jira/browse/HBASE-19970 > Project: HBase > Issue Type: Task >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-19970.master.001.patch > > > Functions deleted in TableAuthManager: > - setTableUserPermissions > - setTableGroupPermissions > - setNamespaceUserPermissions > - setNamespaceGroupPermissions > - writeTableToZooKeeper > - writeNamespaceToZooKeeper > To make sure it was not a bug, and that relevant functionality moved to some > alternate code path, tried to find out why and when these functions went out > of use. But just couldn't figure out...until i reached the patch which added > them. Looks like they were dead functions to start with :) > Jira which added them: HBASE-8409. Commit id: > ac10b3c13d6b66e12d0c9601204b01dfa525ed19 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19966) The WriteEntry for WALKey maybe null if we failed to call WAL.append
[ https://issues.apache.org/jira/browse/HBASE-19966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359218#comment-16359218 ] stack commented on HBASE-19966: --- Failure is in hbase-thrift schema download [ERROR] Failed to execute goal org.apache.maven.plugins:maven-install-plugin:2.5.2:install (default-install) on project hbase-thrift: Failed to install metadata org.apache.hbase:hbase-thrift:3.0.0-SNAPSHOT/maven-metadata.xml: Could not parse metadata /home/jenkins/.m2/repository/org/apache/hbase/hbase-thrift/3.0.0-SNAPSHOT/maven-metadata-local.xml: in epilog non whitespace content is not allowed but got / (position: END_TAG seen ...\n/... @25:2) -> [Help 1] Let me push. > The WriteEntry for WALKey maybe null if we failed to call WAL.append > > > Key: HBASE-19966 > URL: https://issues.apache.org/jira/browse/HBASE-19966 > Project: HBase > Issue Type: Bug > Components: wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19966.patch, HBASE-19966.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19966) The WriteEntry for WALKey maybe null if we failed to call WAL.append
[ https://issues.apache.org/jira/browse/HBASE-19966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19966: -- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to branch-2 and master. Thanks [~Apache9] > The WriteEntry for WALKey maybe null if we failed to call WAL.append > > > Key: HBASE-19966 > URL: https://issues.apache.org/jira/browse/HBASE-19966 > Project: HBase > Issue Type: Bug > Components: wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19966.patch, HBASE-19966.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19950) Introduce a ColumnValueFilter
[ https://issues.apache.org/jira/browse/HBASE-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359224#comment-16359224 ] Chia-Ping Tsai commented on HBASE-19950: 1) Why we don't allow user to pass the null qualifier? The null qualifier is valid in hbase. Perhaps we can convert the passed null qualifier to empty array? {code:java} + public ColumnValueFilter(final byte[] family, final byte[] qualifier, + final CompareOperator op, + final ByteArrayComparable comparator) { + this.family = Preconditions.checkNotNull(family, "family should not be null."); + this.qualifier = Preconditions.checkNotNull(qualifier, "qualifier should not be null"); + this.op = Preconditions.checkNotNull(op, "CompareOperator should not be null"); + this.comparator = Preconditions.checkNotNull(comparator, "Comparator should not be null"); + }{code} 2) ColumnValueFilter doesn't filter the cell by the row, right? {code:java} + @Override + public boolean hasFilterRow() { + return true; + }{code} 3) Seems to me all fields can be final? {code:java} + private byte[] family; + private byte[] qualifier; + private CompareOperator op; + private ByteArrayComparable comparator;{code} 4) Please add the check of NO_OP. Otherwise, the compare will throw the RuntimeException {code:java} + private boolean compareValue(final CompareOperator op, final ByteArrayComparable comparator, + final Cell cell) { + int compareResult = PrivateCellUtil.compareValue(cell, comparator); + return CompareFilter.compare(op, compareResult); + }{code} > Introduce a ColumnValueFilter > - > > Key: HBASE-19950 > URL: https://issues.apache.org/jira/browse/HBASE-19950 > Project: HBase > Issue Type: Improvement > Components: Filters >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Minor > Attachments: HBASE-19950.master.001.patch, > HBASE-19950.master.002.patch, HBASE-19950.master.003.patch > > > Different from {{SingleColumnValueFilter}} which returns an entire row when > specified condition is matched, this new filter will return the matched cell > only. There're already some discussions in HBASE-19824. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19876) The exception happening in converting pb mutation to hbase.mutation messes up the CellScanner
[ https://issues.apache.org/jira/browse/HBASE-19876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359225#comment-16359225 ] Chia-Ping Tsai commented on HBASE-19876: The failed tests pass locally...Try the patch for master. Ping for reviews also. > The exception happening in converting pb mutation to hbase.mutation messes up > the CellScanner > - > > Key: HBASE-19876 > URL: https://issues.apache.org/jira/browse/HBASE-19876 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 1.3.2, 1.5.0, 1.2.7, 2.0.0-beta-2, 1.4.2 > > Attachments: HBASE-19876.branch-1.2.v0.patch, > HBASE-19876.master.001.patch, HBASE-19876.v0.patch, HBASE-19876.v1.patch, > HBASE-19876.v2.patch, HBASE-19876.v3.patch, HBASE-19876.v3.patch, > HBASE-19876.v3.patch, HBASE-19876.v3.patch, HBASE-19876.v4.patch, > HBASE-19876.v5.patch > > > {code:java} > 2018-01-27 22:51:43,794 INFO [hconnection-0x3291b443-shared-pool11-t6] > client.AsyncRequestFutureImpl(778): id=5, table=testQuotaStatusFromMaster3, > attempt=6/16 failed=20ops, last > exception=org.apache.hadoop.hbase.client.WrongRowIOException: > org.apache.hadoop.hbase.client.WrongRowIOException: The row in xxx doesn't > match the original one aaa > at org.apache.hadoop.hbase.client.Mutation.add(Mutation.java:776) > at org.apache.hadoop.hbase.client.Put.add(Put.java:282) > at > org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toPut(ProtobufUtil.java:642) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:952) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:896) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2591) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304){code} > I noticed this bug when testing the table space quota. > When rs are converting pb mutation to hbase.mutation, the quota exception or > cell exception may be thrown. > {code} > Unable to find source-code formatter for language: > rsrpcservices#dobatchop.java. Available languages are: actionscript, ada, > applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, > java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, > rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml for > (ClientProtos.Action action: mutations) { > MutationProto m = action.getMutation(); > Mutation mutation; > if (m.getMutateType() == MutationType.PUT) { > mutation = ProtobufUtil.toPut(m, cells); > batchContainsPuts = true; > } else { > mutation = ProtobufUtil.toDelete(m, cells); > batchContainsDelete = true; > } > mutationActionMap.put(mutation, action); > mArray[i++] = mutation; > checkCellSizeLimit(region, mutation); > // Check if a space quota disallows this mutation > spaceQuotaEnforcement.getPolicyEnforcement(region).check(mutation); > quota.addMutation(mutation); > } > {code} > rs has caught the exception but it doesn't have the cellscanner skip the > failed cells. > {code:java} > } catch (IOException ie) { > if (atomic) { > throw ie; > } > for (Action mutation : mutations) { > builder.addResultOrException(getResultOrException(ie, > mutation.getIndex())); > } > } > {code} > The bug results in the WrongRowIOException to remaining mutations since they > refer to invalid cells. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19876) The exception happening in converting pb mutation to hbase.mutation messes up the CellScanner
[ https://issues.apache.org/jira/browse/HBASE-19876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-19876: --- Attachment: HBASE-19876.v6.patch > The exception happening in converting pb mutation to hbase.mutation messes up > the CellScanner > - > > Key: HBASE-19876 > URL: https://issues.apache.org/jira/browse/HBASE-19876 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 1.3.2, 1.5.0, 1.2.7, 2.0.0-beta-2, 1.4.2 > > Attachments: HBASE-19876.branch-1.2.v0.patch, > HBASE-19876.master.001.patch, HBASE-19876.v0.patch, HBASE-19876.v1.patch, > HBASE-19876.v2.patch, HBASE-19876.v3.patch, HBASE-19876.v3.patch, > HBASE-19876.v3.patch, HBASE-19876.v3.patch, HBASE-19876.v4.patch, > HBASE-19876.v5.patch, HBASE-19876.v6.patch > > > {code:java} > 2018-01-27 22:51:43,794 INFO [hconnection-0x3291b443-shared-pool11-t6] > client.AsyncRequestFutureImpl(778): id=5, table=testQuotaStatusFromMaster3, > attempt=6/16 failed=20ops, last > exception=org.apache.hadoop.hbase.client.WrongRowIOException: > org.apache.hadoop.hbase.client.WrongRowIOException: The row in xxx doesn't > match the original one aaa > at org.apache.hadoop.hbase.client.Mutation.add(Mutation.java:776) > at org.apache.hadoop.hbase.client.Put.add(Put.java:282) > at > org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toPut(ProtobufUtil.java:642) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:952) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:896) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2591) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41560) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304){code} > I noticed this bug when testing the table space quota. > When rs are converting pb mutation to hbase.mutation, the quota exception or > cell exception may be thrown. > {code} > Unable to find source-code formatter for language: > rsrpcservices#dobatchop.java. Available languages are: actionscript, ada, > applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, > java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, > rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml for > (ClientProtos.Action action: mutations) { > MutationProto m = action.getMutation(); > Mutation mutation; > if (m.getMutateType() == MutationType.PUT) { > mutation = ProtobufUtil.toPut(m, cells); > batchContainsPuts = true; > } else { > mutation = ProtobufUtil.toDelete(m, cells); > batchContainsDelete = true; > } > mutationActionMap.put(mutation, action); > mArray[i++] = mutation; > checkCellSizeLimit(region, mutation); > // Check if a space quota disallows this mutation > spaceQuotaEnforcement.getPolicyEnforcement(region).check(mutation); > quota.addMutation(mutation); > } > {code} > rs has caught the exception but it doesn't have the cellscanner skip the > failed cells. > {code:java} > } catch (IOException ie) { > if (atomic) { > throw ie; > } > for (Action mutation : mutations) { > builder.addResultOrException(getResultOrException(ie, > mutation.getIndex())); > } > } > {code} > The bug results in the WrongRowIOException to remaining mutations since they > refer to invalid cells. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19965) Fix flaky TestAsyncRegionAdminApi
[ https://issues.apache.org/jira/browse/HBASE-19965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359226#comment-16359226 ] Guanghao Zhang commented on HBASE-19965: For the regions belong to disabled table or splitted region, we skip to assign it in AssignProcedure.startTransition method. Maybe we can use the same pattern for merged region. Thanks [~stack] for pick up this. :-) {code:java} // Don't assign if table is in disabling of disabled state. TableStateManager tsm = env.getMasterServices().getTableStateManager(); TableName tn = regionNode.getRegionInfo().getTable(); if (tsm.isTableState(tn, TableState.State.DISABLING, TableState.State.DISABLED)) { LOG.info("Table " + tn + " state=" + tsm.getTableState(tn) + ", skipping " + this); return false; } // If the region is SPLIT, we can't assign it. But state might be CLOSED, rather than // SPLIT which is what a region gets set to when Unassigned as part of SPLIT. FIX. if (regionNode.isInState(State.SPLIT) || (regionNode.getRegionInfo().isOffline() && regionNode.getRegionInfo().isSplit())) { LOG.info("SPLIT, cannot be assigned; " + this + "; " + regionNode + "; hri=" + regionNode.getRegionInfo()); return false; } {code} > Fix flaky TestAsyncRegionAdminApi > - > > Key: HBASE-19965 > URL: https://issues.apache.org/jira/browse/HBASE-19965 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-2 > > > See > [https://builds.apache.org/job/HBase%20Nightly/job/branch-2/284/testReport/junit/org.apache.hadoop.hbase.client/TestAsyncRegionAdminApi/testMergeRegions_0_/] > > java.lang.AssertionError: expected:<2> but was:<3> at > org.apache.hadoop.hbase.client.TestAsyncRegionAdminApi.testMergeRegions(TestAsyncRegionAdminApi.java:359) > > Merge regions not work. The table still have 3 regions after the > MergeRegionsProcedure finished. > The master start balance region 9e2773ba1efba79a2defa276e9a26ed4. But because > the MergeRegionsProcedure pid=138 start work first, so the balance need wait > for the lock. But after merge regions finished, the MoveRegionProcedure > pid=139 start work and assign 9e2773ba1efba79a2defa276e9a26ed4 to a new > region server. This is not right. The MoveRegionProcedure should skip to > assign a region which was marked as offline. Or we should clear the merged > regions' procedure when MergeRegionsProcedure finished. > > Logs: > 2018-02-08 16:24:44,608 INFO [master/cd4730e3eae2:0.Chore.1] > master.HMaster(1454): balance > hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., > source=cd4730e3eae2,39077,1518106776411, > destination=cd4730e3eae2,40578,1518106776318 > 2018-02-08 16:24:44,608 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37885] > procedure2.ProcedureExecutor(868): Stored pid=138, > state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE; MergeTableRegionsProcedure > table=testMergeRegions, regions=[9e2773ba1efba79a2defa276e9a26ed4, > 8f8fd5cd032313e1aadb83e31e1b7479], forcibly=false > .. > 2018-02-08 16:24:50,111 INFO [PEWorker-13] > procedure2.ProcedureExecutor(1249): Finished pid=138, state=SUCCESS; > MergeTableRegionsProcedure table=testMergeRegions, > regions=[9e2773ba1efba79a2defa276e9a26ed4, 8f8fd5cd032313e1aadb83e31e1b7479], > forcibly=false in 5.5710sec > 2018-02-08 16:24:50,113 INFO [PEWorker-13] > procedure.MasterProcedureScheduler(813): pid=139, > state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure > hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., > source=cd4730e3eae2,39077,1518106776411, > destination=cd4730e3eae2,40578,1518106776318 testMergeRegions > testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19966) The WriteEntry for WALKey maybe null if we failed to call WAL.append
[ https://issues.apache.org/jira/browse/HBASE-19966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359230#comment-16359230 ] Duo Zhang commented on HBASE-19966: --- Thanks for taking care of this sir. > The WriteEntry for WALKey maybe null if we failed to call WAL.append > > > Key: HBASE-19966 > URL: https://issues.apache.org/jira/browse/HBASE-19966 > Project: HBase > Issue Type: Bug > Components: wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19966.patch, HBASE-19966.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-16060) 1.x clients cannot access table state talking to 2.0 cluster
[ https://issues.apache.org/jira/browse/HBASE-16060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16060: -- Attachment: HBASE-16060.branch-2.003.patch > 1.x clients cannot access table state talking to 2.0 cluster > > > Key: HBASE-16060 > URL: https://issues.apache.org/jira/browse/HBASE-16060 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: stack >Priority: Blocker > Fix For: 2.0.0-beta-2 > > Attachments: > 0002-HBASE-16060-1.x-clients-cannot-access-table-state-ta.patch, > HBASE-16060.branch-2.001.patch, HBASE-16060.branch-2.002.patch, > HBASE-16060.branch-2.003.patch > > > Since table state is migrated to meta instead of zk in 2.0, 1.x clients > talking to 2.0 cluster cannot access the table state. This causes some weird > behavior since from a client perspective, {{Admin.isTableEnabled()}} and > {{Admin.isTableDisabled()}} both return false. > One option we can do is to add code in 1.x clients so that they can access > the table state in meta if needed. Otherwise, we can mirror the table state > in zk (while keeping meta as the source of truth) during 2.x lifecycle so > that any 1.x client can still work correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19965) Fix flaky TestAsyncRegionAdminApi
[ https://issues.apache.org/jira/browse/HBASE-19965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359234#comment-16359234 ] stack commented on HBASE-19965: --- Thank you [~zghaobac] Let me look tomorrow. I came across another instance today where on restart, if CLOSED, we presume an outstanding assign though the table is disabled This seems similar. Let me study it. Thanks for the research. > Fix flaky TestAsyncRegionAdminApi > - > > Key: HBASE-19965 > URL: https://issues.apache.org/jira/browse/HBASE-19965 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-2 > > > See > [https://builds.apache.org/job/HBase%20Nightly/job/branch-2/284/testReport/junit/org.apache.hadoop.hbase.client/TestAsyncRegionAdminApi/testMergeRegions_0_/] > > java.lang.AssertionError: expected:<2> but was:<3> at > org.apache.hadoop.hbase.client.TestAsyncRegionAdminApi.testMergeRegions(TestAsyncRegionAdminApi.java:359) > > Merge regions not work. The table still have 3 regions after the > MergeRegionsProcedure finished. > The master start balance region 9e2773ba1efba79a2defa276e9a26ed4. But because > the MergeRegionsProcedure pid=138 start work first, so the balance need wait > for the lock. But after merge regions finished, the MoveRegionProcedure > pid=139 start work and assign 9e2773ba1efba79a2defa276e9a26ed4 to a new > region server. This is not right. The MoveRegionProcedure should skip to > assign a region which was marked as offline. Or we should clear the merged > regions' procedure when MergeRegionsProcedure finished. > > Logs: > 2018-02-08 16:24:44,608 INFO [master/cd4730e3eae2:0.Chore.1] > master.HMaster(1454): balance > hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., > source=cd4730e3eae2,39077,1518106776411, > destination=cd4730e3eae2,40578,1518106776318 > 2018-02-08 16:24:44,608 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37885] > procedure2.ProcedureExecutor(868): Stored pid=138, > state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE; MergeTableRegionsProcedure > table=testMergeRegions, regions=[9e2773ba1efba79a2defa276e9a26ed4, > 8f8fd5cd032313e1aadb83e31e1b7479], forcibly=false > .. > 2018-02-08 16:24:50,111 INFO [PEWorker-13] > procedure2.ProcedureExecutor(1249): Finished pid=138, state=SUCCESS; > MergeTableRegionsProcedure table=testMergeRegions, > regions=[9e2773ba1efba79a2defa276e9a26ed4, 8f8fd5cd032313e1aadb83e31e1b7479], > forcibly=false in 5.5710sec > 2018-02-08 16:24:50,113 INFO [PEWorker-13] > procedure.MasterProcedureScheduler(813): pid=139, > state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure > hri=testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4., > source=cd4730e3eae2,39077,1518106776411, > destination=cd4730e3eae2,40578,1518106776318 testMergeRegions > testMergeRegions,,1518107079782.9e2773ba1efba79a2defa276e9a26ed4. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19920) TokenUtil.obtainToken unnecessarily creates a local directory
[ https://issues.apache.org/jira/browse/HBASE-19920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359248#comment-16359248 ] Mike Drob commented on HBASE-19920: --- Not sure why it couldn't comment... https://builds.apache.org/job/PreCommit-HBASE-Build/11457 ran against v8 {noformat} 02:35:47.560 02:35:47.560 02:35:47.560 Adding comment to JIRA 02:35:47.560 02:35:47.560 02:35:47.560 02:35:47.560 02:37:11.642 02:37:11.642 02:37:11.642 +1 overall 02:37:11.642 02:37:11.764 | Vote | Subsystem | Runtime | Comment 02:37:11.764 02:37:11.822 | 0 | reexec | 0m 12s | Docker mode activated. 02:37:11.853 | | || Prechecks 02:37:11.883 | 0 | findbugs | 0m 0s | Findbugs executables are not available. 02:37:11.910 | +1 | hbaseanti | 0m 0s | Patch does not have any anti-patterns. 02:37:11.939 | +1 |@author | 0m 0s | The patch does not contain any @author 02:37:11.939 | | || tags. 02:37:11.967 | +1 | test4tests | 0m 0s | The patch appears to include 1 new or 02:37:11.967 | | || modified test files. 02:37:11.994 | | || master Compile Tests 02:37:12.026 | 0 | mvndep | 0m 23s | Maven dependency ordering for branch 02:37:12.061 | +1 | mvninstall | 4m 25s | master passed 02:37:12.091 | +1 |compile | 1m 3s | master passed 02:37:12.116 | +1 | checkstyle | 1m 39s | master passed 02:37:12.145 | +1 | shadedjars | 6m 21s | branch has no errors when building our 02:37:12.145 | | || shaded downstream artifacts. 02:37:12.173 | +1 |javadoc | 0m 48s | master passed 02:37:12.197 | | || Patch Compile Tests 02:37:12.221 | 0 | mvndep | 0m 14s | Maven dependency ordering for patch 02:37:12.252 | +1 | mvninstall | 4m 21s | the patch passed 02:37:12.281 | +1 |compile | 1m 2s | the patch passed 02:37:12.307 | +1 | javac | 1m 2s | the patch passed 02:37:12.339 | +1 | checkstyle | 1m 37s | the patch passed 02:37:12.372 | +1 | whitespace | 0m 0s | The patch has no whitespace issues. 02:37:12.401 | +1 | shadedjars | 4m 42s | patch has no errors when building our 02:37:12.401 | | || shaded downstream artifacts. 02:37:12.437 | +1 |hadoopcheck | 18m 58s | Patch does not cause any errors with 02:37:12.437 | | || Hadoop 2.6.5 2.7.4 or 3.0.0. 02:37:12.480 | +1 |javadoc | 0m 50s | the patch passed 02:37:12.509 | | || Other Tests 02:37:12.538 | +1 | unit | 3m 4s | hbase-client in the patch passed. 02:37:12.563 | +1 | unit | 111m 12s | hbase-server in the patch passed. 02:37:12.593 | +1 | asflicense | 0m 40s | The patch does not generate ASF License 02:37:12.593 | | || warnings. 02:37:12.625 | | | 155m 36s | 02:37:12.626 02:37:12.626 02:37:12.626 || Subsystem || Report/Notes || 02:37:12.626 02:37:12.632 | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 | 02:37:12.637 | JIRA Issue | HBASE-19920 | 02:37:12.642 | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12909862/HBASE-19920.v8.patch | 02:37:12.647 | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | 02:37:12.652 | uname | Linux 3b98d43f13bd 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 GNU/Linux | 02:37:12.656 | Build tool | maven | 02:37:12.660 | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | 02:37:12.663 | git revision | master / f1a81618fd | 02:37:12.688 | maven | version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) | 02:37:12.688 | Default Java | 1.8.0_151 | 02:37:12.688 | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/11457/testReport/ | 02:37:12.688 | Max. process+thread count | 4404 (vs. ulimit of 1) | 02:37:12.688 | modules | C: hbase-client
[jira] [Commented] (HBASE-19970) Remove unused functions from TableAuthManager
[ https://issues.apache.org/jira/browse/HBASE-19970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359265#comment-16359265 ] Hadoop QA commented on HBASE-19970: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m 2s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 59s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 10s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m 56s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 31s{color} | {color:red} hbase-server: The patch generated 3 new + 89 unchanged - 12 fixed = 92 total (was 101) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m 10s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 23m 59s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}104m 12s{color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 31s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}151m 5s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 | | JIRA Issue | HBASE-19970 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12910015/HBASE-19970.master.001.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 23045f8699f1 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 15:49:21 UTC 2017 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 2b63af376e | | maven | version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) | | Default Java | 1.8.0_151 | | checkstyle | https://builds.apache.org/job/PreCommit-HBASE-Build/11472/artifact/patchprocess/diff-checkstyle-hbase-server.txt | | unit | https://builds.apache.org/job/PreCommit-HBASE-Build/11472/artifact/patchprocess/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/11472/testReport/ | | Max. process+thread count | 4831 (
[jira] [Commented] (HBASE-19876) The exception happening in converting pb mutation to hbase.mutation messes up the CellScanner
[ https://issues.apache.org/jira/browse/HBASE-19876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359276#comment-16359276 ] Hadoop QA commented on HBASE-19876: --- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 19s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 36s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m 17s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 49s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 19m 14s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 1s{color} | {color:green} hbase-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green}115m 15s{color} | {color:green} hbase-server in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 51s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}160m 1s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 | | JIRA Issue | HBASE-19876 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12910018/HBASE-19876.v6.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 7387a76c9d23 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 32f235abee | | maven | version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) | | Default Java | 1.8.0_151 | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/11473/testReport/ | | Max. process+thread c
[jira] [Commented] (HBASE-16060) 1.x clients cannot access table state talking to 2.0 cluster
[ https://issues.apache.org/jira/browse/HBASE-16060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359280#comment-16359280 ] Hadoop QA commented on HBASE-16060: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} branch-2 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 27s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 34s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 51s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m 1s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s{color} | {color:green} branch-2 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 9s{color} | {color:green} The patch hbase-protocol-shaded passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 32s{color} | {color:green} hbase-client: The patch generated 0 new + 26 unchanged - 1 fixed = 26 total (was 27) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 14s{color} | {color:red} hbase-server: The patch generated 10 new + 294 unchanged - 5 fixed = 304 total (was 299) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 7s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 14m 55s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 29s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 2s{color} | {color:green} hbase-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green}115m 33s{color} | {color:green} hbase-server in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 56s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}157m 16s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker |
[jira] [Updated] (HBASE-19950) Introduce a ColumnValueFilter
[ https://issues.apache.org/jira/browse/HBASE-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-19950: -- Attachment: HBASE-19950.master.004.patch > Introduce a ColumnValueFilter > - > > Key: HBASE-19950 > URL: https://issues.apache.org/jira/browse/HBASE-19950 > Project: HBase > Issue Type: Improvement > Components: Filters >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Minor > Attachments: HBASE-19950.master.001.patch, > HBASE-19950.master.002.patch, HBASE-19950.master.003.patch, > HBASE-19950.master.004.patch > > > Different from {{SingleColumnValueFilter}} which returns an entire row when > specified condition is matched, this new filter will return the matched cell > only. There're already some discussions in HBASE-19824. -- This message was sent by Atlassian JIRA (v7.6.3#76005)