[jira] [Commented] (HBASE-19726) Failed to start HMaster due to infinite retrying on meta assign
[
https://issues.apache.org/jira/browse/HBASE-19726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351322#comment-16351322
]
Hudson commented on HBASE-19726:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4517 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/4517/])
HBASE-19726 Failed to start HMaster due to infinite retrying on meta (stack:
rev 41974efa85a73f1967fc57c8e7df7b1344d56714)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
> Failed to start HMaster due to infinite retrying on meta assign
> ---
>
> Key: HBASE-19726
> URL: https://issues.apache.org/jira/browse/HBASE-19726
> Project: HBase
> Issue Type: Bug
>Reporter: Duo Zhang
>Assignee: stack
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19726.patch
>
>
> This is what I got at first, an exception when trying to write something to
> meta when meta has not been onlined yet.
> {noformat}
> 2018-01-07,21:03:14,389 INFO org.apache.hadoop.hbase.master.HMaster: Running
> RecoverMetaProcedure to ensure proper hbase:meta deploy.
> 2018-01-07,21:03:14,637 INFO
> org.apache.hadoop.hbase.master.procedure.RecoverMetaProcedure: Start pid=1,
> state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure
> failedMetaServer=null, splitWal=true
> 2018-01-07,21:03:14,645 INFO org.apache.hadoop.hbase.master.MasterWalManager:
> Log folder
> hdfs://c402tst-community/hbase/c402tst-community/WALs/c4-hadoop-tst-st27.bj,38900,1515330173896
> belongs to an existing region server
> 2018-01-07,21:03:14,646 INFO org.apache.hadoop.hbase.master.MasterWalManager:
> Log folder
> hdfs://c402tst-community/hbase/c402tst-community/WALs/c4-hadoop-tst-st29.bj,38900,1515330177232
> belongs to an existing region server
> 2018-01-07,21:03:14,648 INFO
> org.apache.hadoop.hbase.master.procedure.RecoverMetaProcedure: pid=1,
> state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure
> failedMetaServer=null, splitWal=true; Retaining meta assignment to server=null
> 2018-01-07,21:03:14,653 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Initialized
> subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE;
> AssignProcedure table=hbase:meta, region=1588230740}]
> 2018-01-07,21:03:14,660 INFO
> org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler: pid=2,
> ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
> table=hbase:meta, region=1588230740 hbase:meta hbase:meta,,1.1588230740
> 2018-01-07,21:03:14,663 INFO
> org.apache.hadoop.hbase.master.assignment.AssignProcedure: Start pid=2,
> ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
> table=hbase:meta, region=1588230740; rit=OFFLINE, location=null;
> forceNewPlan=false, retain=false
> 2018-01-07,21:03:14,831 INFO
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator: Setting hbase:meta
> (replicaId=0) location in ZooKeeper as
> c4-hadoop-tst-st27.bj,38900,1515330173896
> 2018-01-07,21:03:14,841 INFO
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure: Dispatch
> pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure
> table=hbase:meta, region=1588230740; rit=OPENING,
> location=c4-hadoop-tst-st27.bj,38900,1515330173896
> 2018-01-07,21:03:14,992 INFO
> org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher: Using
> procedure batch rpc execution for
> serverName=c4-hadoop-tst-st27.bj,38900,1515330173896 version=3145728
> 2018-01-07,21:03:15,593 ERROR
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: Cannot get replica 0
> location for
> {"totalColumns":1,"row":"hbase:meta","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515330195514}]},"ts":1515330195514}
> 2018-01-07,21:03:15,594 WARN
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure:
> Retryable error trying to transition: pid=2, ppid=1,
> state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=hbase:meta,
> region=1588230740; rit=OPEN,
> location=c4-hadoop-tst-st27.bj,38900,1515330173896
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: IOException: 1 time, servers with issues: null
> at
> org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:54)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.getErrors(AsyncRequestFutureImpl.java:1250)
> at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:457)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:570)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.put(MetaTableAccessor.java:1450)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.putToMetaTable(MetaTableAccessor.java:14
[jira] [Commented] (HBASE-19915) From split/ merge procedures daughter/ merged regions get created in OFFLINE state
[ https://issues.apache.org/jira/browse/HBASE-19915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351321#comment-16351321 ] Hudson commented on HBASE-19915: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4517 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4517/]) HBASE-19915 Create merged/ daughter region/s with initial state CLOSED (stack: rev 811afad103dacfe50788f08b41d804e216af5afc) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java > From split/ merge procedures daughter/ merged regions get created in OFFLINE > state > -- > > Key: HBASE-19915 > URL: https://issues.apache.org/jira/browse/HBASE-19915 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0-beta-1 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: hbase-19915.master.001.patch, > hbase-19915.master.001.patch > > > See HBASE-19530. When regions are created initial state should be CLOSED. Bug > was discovered while debugging flaky test > TestSplitTableRegionProcedure#testRollbackAndDoubleExecution with numOfSteps > set to 4. After updating daughter regions in meta when master is restarted, > startup sequence of master assigns all OFFLINE regions. As daughter regions > are stored with OFFLINE state, daughter regions are assigned. This is > followed by re-assignment of daughter regions from resumed > SplitTableRegionProcedure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19904) Break dependency of WAL constructor on Replication
[ https://issues.apache.org/jira/browse/HBASE-19904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351329#comment-16351329 ] Duo Zhang commented on HBASE-19904: --- Thanks [~appy], will commit shortly. On the inner class for wal listener, ReplicationSourceManager has already been large enough so I do not want to add another stuff into it. Maybe a separated class? What do you think? Anyway, let me open an issue for it. > Break dependency of WAL constructor on Replication > -- > > Key: HBASE-19904 > URL: https://issues.apache.org/jira/browse/HBASE-19904 > Project: HBase > Issue Type: Improvement > Components: Replication, wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19904-branch-2.patch, HBASE-19904-branch-2.patch, > HBASE-19904-v3.patch, HBASE-19904-v3.patch, HBASE-19904-v4.patch, > HBASE-19904-v4.patch, HBASE-19904-v5.patch > > > When implementing synchronous replication, I found that we need to depend > more on replication in WAL so it is even more pain... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19904) Break dependency of WAL constructor on Replication
[ https://issues.apache.org/jira/browse/HBASE-19904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351337#comment-16351337 ] Duo Zhang commented on HBASE-19904: --- Oh just saw your last comments. So you also think RSM is big enough, good. Let me open an issue to make the listener a separated class. > Break dependency of WAL constructor on Replication > -- > > Key: HBASE-19904 > URL: https://issues.apache.org/jira/browse/HBASE-19904 > Project: HBase > Issue Type: Improvement > Components: Replication, wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19904-branch-2.patch, HBASE-19904-branch-2.patch, > HBASE-19904-v3.patch, HBASE-19904-v3.patch, HBASE-19904-v4.patch, > HBASE-19904-v4.patch, HBASE-19904-v5.patch > > > When implementing synchronous replication, I found that we need to depend > more on replication in WAL so it is even more pain... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19904) Break dependency of WAL constructor on Replication
[ https://issues.apache.org/jira/browse/HBASE-19904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19904: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master and branch-2. Thanks all for reviewing. > Break dependency of WAL constructor on Replication > -- > > Key: HBASE-19904 > URL: https://issues.apache.org/jira/browse/HBASE-19904 > Project: HBase > Issue Type: Improvement > Components: Replication, wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19904-branch-2.patch, HBASE-19904-branch-2.patch, > HBASE-19904-v3.patch, HBASE-19904-v3.patch, HBASE-19904-v4.patch, > HBASE-19904-v4.patch, HBASE-19904-v5.patch > > > When implementing synchronous replication, I found that we need to depend > more on replication in WAL so it is even more pain... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19919) Tidying up logging
[ https://issues.apache.org/jira/browse/HBASE-19919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351341#comment-16351341 ] Hudson commented on HBASE-19919: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4518 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4518/]) HBASE-19919 Tidying up logging (stack: rev 40250f8c5f8efe3016f0c3300d9728b0ee2056c6) * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationPeerConfigUpgrader.java.rej * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java.rej * (add) hbase-common/src/test/java/org/apache/hadoop/hbase/net/TestAddress.java > Tidying up logging > -- > > Key: HBASE-19919 > URL: https://issues.apache.org/jira/browse/HBASE-19919 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: 0001-HBASE-19919-Tidying-up-logging.patch, > HBASE-19919.branch-2.001.patch > > > Reading logs, there is a bunch of stuff we don't need, thread names are too > long, etc. Doing a little tidying. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19528) Major Compaction Tool
[
https://issues.apache.org/jira/browse/HBASE-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351343#comment-16351343
]
Hadoop QA commented on HBASE-19528:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m
46s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
0s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} branch-1 passed with JDK v1.8.0_162 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} branch-1 passed with JDK v1.7.0_171 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
52s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} branch-1 passed with JDK v1.8.0_162 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} branch-1 passed with JDK v1.7.0_171 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed with JDK v1.8.0_162 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed with JDK v1.7.0_171 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
17s{color} | {color:red} hbase-server: The patch generated 8 new + 0 unchanged
- 0 fixed = 8 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
32s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m
29s{color} | {color:red} The patch causes 44 errors with Hadoop v2.4.1. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m
23s{color} | {color:red} The patch causes 44 errors with Hadoop v2.5.2. {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed with JDK v1.8.0_162 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed with JDK v1.7.0_171 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}103m
17s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}143m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:36a7029 |
| JIRA Issue | HBASE-19528 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12909086/HB
[jira] [Commented] (HBASE-19528) Major Compaction Tool
[
https://issues.apache.org/jira/browse/HBASE-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351344#comment-16351344
]
Hadoop QA commented on HBASE-19528:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
4s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
0s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} branch-1 passed with JDK v1.8.0_162 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} branch-1 passed with JDK v1.7.0_171 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
52s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} branch-1 passed with JDK v1.8.0_162 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} branch-1 passed with JDK v1.7.0_171 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed with JDK v1.8.0_162 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed with JDK v1.7.0_171 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
17s{color} | {color:red} hbase-server: The patch generated 8 new + 0 unchanged
- 0 fixed = 8 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
34s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m
30s{color} | {color:red} The patch causes 44 errors with Hadoop v2.4.1. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m
25s{color} | {color:red} The patch causes 44 errors with Hadoop v2.5.2. {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed with JDK v1.8.0_162 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed with JDK v1.7.0_171 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}103m
34s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}141m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:36a7029 |
| JIRA Issue | HBASE-19528 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12909086/HB
[jira] [Created] (HBASE-19926) Use a separated class to implement the WALActionListener for Replication
Duo Zhang created HBASE-19926: - Summary: Use a separated class to implement the WALActionListener for Replication Key: HBASE-19926 URL: https://issues.apache.org/jira/browse/HBASE-19926 Project: HBase Issue Type: Bug Components: Replication, wal Reporter: Duo Zhang Assignee: Duo Zhang Fix For: 2.0.0-beta-2 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19826) Provide a option to see rows behind a delete in a time range queries
[
https://issues.apache.org/jira/browse/HBASE-19826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351353#comment-16351353
]
Duo Zhang commented on HBASE-19826:
---
So what if a compaction happens before you run the tool? The cell will be
deleted in compaction if you do not have KeepDeleteCells.TRUE, then either you
use time range scan or not you can never read it anymore...
> Provide a option to see rows behind a delete in a time range queries
>
>
> Key: HBASE-19826
> URL: https://issues.apache.org/jira/browse/HBASE-19826
> Project: HBase
> Issue Type: Improvement
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 2.0.0
>
>
> We can provide an option (something like seePastDeleteMarkers) in a scan to
> let the user see the versions behind the delete marker even if
> keepDeletedCells is set to false in the descriptor.
> With the previous version, we workaround the same in preStoreScannerOpen
> hook. For reference PHOENIX-4277
> {code}
> @Override
> public KeyValueScanner preStoreScannerOpen(final
> ObserverContext c,
> final Store store, final Scan scan, final NavigableSet
> targetCols,
> final KeyValueScanner s) throws IOException {
>
> if (scan.isRaw() ||
> ScanInfoUtil.isKeepDeletedCells(store.getScanInfo()) ||
> scan.getTimeRange().getMax() == HConstants.LATEST_TIMESTAMP ||
> TransactionUtil.isTransactionalTimestamp(scan.getTimeRange().getMax())) {
> return s;
> }
>
> ScanInfo scanInfo =
> ScanInfoUtil.cloneScanInfoWithKeepDeletedCells(store.getScanInfo());
> return new StoreScanner(store, scanInfo, scan, targetCols,
>
> c.getEnvironment().getRegion().getReadpoint(scan.getIsolationLevel()));
> }
> {code}
> Another way is to provide a way to set KEEP_DELETED_CELLS to true in
> ScanOptions of preStoreScannerOpen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19919) Tidying up logging
[ https://issues.apache.org/jira/browse/HBASE-19919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351357#comment-16351357 ] Duo Zhang commented on HBASE-19919: --- Wrong patch sir? The compilation of master is broken... https://github.com/apache/hbase/commit/40250f8c5f8efe3016f0c3300d9728b0ee2056c6 Only 3 files? > Tidying up logging > -- > > Key: HBASE-19919 > URL: https://issues.apache.org/jira/browse/HBASE-19919 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: 0001-HBASE-19919-Tidying-up-logging.patch, > HBASE-19919.branch-2.001.patch > > > Reading logs, there is a bunch of stuff we don't need, thread names are too > long, etc. Doing a little tidying. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (HBASE-19919) Tidying up logging
[ https://issues.apache.org/jira/browse/HBASE-19919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reopened HBASE-19919: --- Revert the patch for master for now. I think you got a conflict when applying the patch to master [~stack], buy you still pushed it... There are two 'rej' files in the commit... > Tidying up logging > -- > > Key: HBASE-19919 > URL: https://issues.apache.org/jira/browse/HBASE-19919 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: 0001-HBASE-19919-Tidying-up-logging.patch, > HBASE-19919.branch-2.001.patch > > > Reading logs, there is a bunch of stuff we don't need, thread names are too > long, etc. Doing a little tidying. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19900) Region-level exception destroy the result of batch
[ https://issues.apache.org/jira/browse/HBASE-19900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-19900: --- Attachment: HBASE-19900.v0.patch > Region-level exception destroy the result of batch > -- > > Key: HBASE-19900 > URL: https://issues.apache.org/jira/browse/HBASE-19900 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 1.3.2, 1.5.0, 1.2.7, 2.0.0-beta-2, 1.4.2 > > Attachments: HBASE-19900.v0.patch > > > 1) decrease action count repeatedly > If the AsyncRequestFuture#waitUntilDone return prematurely, user will get the > incorrect results. Or user will be block by AsyncRequestFuture#waitUntilDone > as the count is never equal with 0. > 2) the successive result will be overwrited > 3) The failed op is added to RetriesExhaustedWithDetailsException repeatedly > AsyncRequestFutureImpl#receiveMultiAction process the action-lever error > first, and then add the region-level exception to each action. Hence, user > may get the various exceptions for the same action (row op) from the > RetriesExhaustedWithDetailsException. > In fact, if both of action-level exception and region-lever exception exist, > they always have the same context. I'm not sure whether that is what > RetriesExhaustedWithDetailsException want. As i see it, we shouldn't have the > duplicate ops in RetriesExhaustedWithDetailsException since that may confuse > users if they catch the RetriesExhaustedWithDetailsException to check the > invalid operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19900) Region-level exception destroy the result of batch
[ https://issues.apache.org/jira/browse/HBASE-19900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-19900: --- Status: Patch Available (was: Open) > Region-level exception destroy the result of batch > -- > > Key: HBASE-19900 > URL: https://issues.apache.org/jira/browse/HBASE-19900 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 1.3.2, 1.5.0, 1.2.7, 2.0.0-beta-2, 1.4.2 > > Attachments: HBASE-19900.v0.patch > > > 1) decrease action count repeatedly > If the AsyncRequestFuture#waitUntilDone return prematurely, user will get the > incorrect results. Or user will be block by AsyncRequestFuture#waitUntilDone > as the count is never equal with 0. > 2) the successive result will be overwrited > 3) The failed op is added to RetriesExhaustedWithDetailsException repeatedly > AsyncRequestFutureImpl#receiveMultiAction process the action-lever error > first, and then add the region-level exception to each action. Hence, user > may get the various exceptions for the same action (row op) from the > RetriesExhaustedWithDetailsException. > In fact, if both of action-level exception and region-lever exception exist, > they always have the same context. I'm not sure whether that is what > RetriesExhaustedWithDetailsException want. As i see it, we shouldn't have the > duplicate ops in RetriesExhaustedWithDetailsException since that may confuse > users if they catch the RetriesExhaustedWithDetailsException to check the > invalid operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19900) Region-level exception destroy the result of batch
[ https://issues.apache.org/jira/browse/HBASE-19900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351372#comment-16351372 ] Chia-Ping Tsai commented on HBASE-19900: The patch merge the check of action-level exception and region-lever exception. If the action fail without any exception, we assign the region-level exception to it. If no region-level exception exists, we assign a unknown error instead. In short, each action won't be handled repeatedly so the following issues should be fixed. # decrease action count repeatedly # the successive result will be overwrited # The failed op is added to RetriesExhaustedWithDetailsException repeatedly The patch also add some debug info to trace the overwrite of result and exception. > Region-level exception destroy the result of batch > -- > > Key: HBASE-19900 > URL: https://issues.apache.org/jira/browse/HBASE-19900 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 1.3.2, 1.5.0, 1.2.7, 2.0.0-beta-2, 1.4.2 > > Attachments: HBASE-19900.v0.patch > > > 1) decrease action count repeatedly > If the AsyncRequestFuture#waitUntilDone return prematurely, user will get the > incorrect results. Or user will be block by AsyncRequestFuture#waitUntilDone > as the count is never equal with 0. > 2) the successive result will be overwrited > 3) The failed op is added to RetriesExhaustedWithDetailsException repeatedly > AsyncRequestFutureImpl#receiveMultiAction process the action-lever error > first, and then add the region-level exception to each action. Hence, user > may get the various exceptions for the same action (row op) from the > RetriesExhaustedWithDetailsException. > In fact, if both of action-level exception and region-lever exception exist, > they always have the same context. I'm not sure whether that is what > RetriesExhaustedWithDetailsException want. As i see it, we shouldn't have the > duplicate ops in RetriesExhaustedWithDetailsException since that may confuse > users if they catch the RetriesExhaustedWithDetailsException to check the > invalid operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19553) Old replica regions should be cleared from AM memory after primary region split or merge
[ https://issues.apache.org/jira/browse/HBASE-19553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351376#comment-16351376 ] Pankaj Kumar commented on HBASE-19553: -- Sorry, have no idea about the Yetus bug, can you please give some example. > Old replica regions should be cleared from AM memory after primary region > split or merge > > > Key: HBASE-19553 > URL: https://issues.apache.org/jira/browse/HBASE-19553 > Project: HBase > Issue Type: Bug > Components: master >Reporter: huaxiang sun >Assignee: Pankaj Kumar >Priority: Minor > Fix For: 1.5.0 > > Attachments: HBASE-19553-branch-1-v2.patch, > HBASE-19553-branch-1-v3.patch, HBASE-19553-branch-1-v4.patch, > HBASE-19553-branch-1.patch > > > Similar to HBASE-18025, the replica parent's info is not removed from master. > Actually I think it can be removed after replica region is split or merged, I > will check the logic and apply one patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19900) Region-level exception destroy the result of batch
[ https://issues.apache.org/jira/browse/HBASE-19900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351377#comment-16351377 ] Chia-Ping Tsai commented on HBASE-19900: All bugs can be easily reproduced by batching the RowMutations. However, only 1.4+ support to do the batch of RowMutations (HBASE-19096). I am not sure whether any real use case can be used to reproduce the bug for branch-1.2 and branch-1.3. > Region-level exception destroy the result of batch > -- > > Key: HBASE-19900 > URL: https://issues.apache.org/jira/browse/HBASE-19900 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 1.3.2, 1.5.0, 1.2.7, 2.0.0-beta-2, 1.4.2 > > Attachments: HBASE-19900.v0.patch > > > 1) decrease action count repeatedly > If the AsyncRequestFuture#waitUntilDone return prematurely, user will get the > incorrect results. Or user will be block by AsyncRequestFuture#waitUntilDone > as the count is never equal with 0. > 2) the successive result will be overwrited > 3) The failed op is added to RetriesExhaustedWithDetailsException repeatedly > AsyncRequestFutureImpl#receiveMultiAction process the action-lever error > first, and then add the region-level exception to each action. Hence, user > may get the various exceptions for the same action (row op) from the > RetriesExhaustedWithDetailsException. > In fact, if both of action-level exception and region-lever exception exist, > they always have the same context. I'm not sure whether that is what > RetriesExhaustedWithDetailsException want. As i see it, we shouldn't have the > duplicate ops in RetriesExhaustedWithDetailsException since that may confuse > users if they catch the RetriesExhaustedWithDetailsException to check the > invalid operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19925) Delete an unreachable peer will triggers all regionservers abort
[
https://issues.apache.org/jira/browse/HBASE-19925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351387#comment-16351387
]
Ted Yu commented on HBASE-19925:
{code}
if ((ke instanceof ConnectionLossException || ke instanceof
SessionExpiredException
70 || ke instanceof AuthFailedException) && this.isRunning()) {
{code}
Can you move the isRunning() condition as the first component in the if
statement ?
Add a comment for the second change.
Is it possible to add a test for this scenario ?
> Delete an unreachable peer will triggers all regionservers abort
>
>
> Key: HBASE-19925
> URL: https://issues.apache.org/jira/browse/HBASE-19925
> Project: HBase
> Issue Type: Bug
>Reporter: Yun Zhao
>Assignee: Yun Zhao
>Priority: Critical
> Attachments: HBASE-19925.master.001.patch
>
>
> Add an unreachable peer
> {code:java}
> add_peer '4', CLUSTER_KEY => "server1.cie.com:2181:/hbase"{code}
> After a while to delete it,Regionserver will appear in the following log and
> stop.
> {code:java}
> 2018-02-02 20:04:25,959 INFO [main-EventThread.replicationSource,4]
> regionserver.ReplicationSource: Replicating
> 5467de52-dc46-45be-902c-110dd7a83e06 -> null
> 2018-02-02 20:04:25,960 ERROR
> [main-EventThread.replicationSource,4.replicationSource..com%2C16020%2C1515498473547.default,4]
> regionserver.ReplicationSource: Unexpected exception in
> ReplicationSourceWorkerThread, currentPath=null
> java.lang.IllegalArgumentException: Peer with id= 4 is not connected
> at
> org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.getStatusOfPeer(ReplicationPeersZKImpl.java:207)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.isPeerEnabled(ReplicationSource.java:327)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceWorkerThread.run(ReplicationSource.java:512)
> 2018-02-02 20:04:25,960 INFO
> [main-EventThread.replicationSource,4.replicationSource..com%2C16020%2C1515498473547.default,4]
> regionserver.HRegionServer: STOPPED: Unexpected exception in
> ReplicationSourceWorkerThread{code}
>
> HBase 1.2.6
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19925) Delete an unreachable peer will triggers all regionservers abort
[
https://issues.apache.org/jira/browse/HBASE-19925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351388#comment-16351388
]
Duo Zhang commented on HBASE-19925:
---
I think we should rewrite the get peerClusterId part, for now it is a bit
complicated as you add another check...
{code}
if (!this.isSourceActive()) {
return;
}
sleepMultiplier = 1;
// delay this until we are in an asynchronous thread
while (this.isSourceActive() && this.peerClusterId == null) {
this.peerClusterId = replicationEndpoint.getPeerUUID();
if (this.isSourceActive() && this.peerClusterId == null) {
if (sleepForRetries("Cannot contact the peer's zk ensemble",
sleepMultiplier)) {
sleepMultiplier++;
}
}
}
{code}
Change to
{code}
sleepMultiplier = 1;
for (;;) {
peerClusterId = XXX;
if (peerClusterId != null) {
break;
}
if (!isSourceActive()) {
return;
}
sleep();
}
{code}
What do you think?
Thanks.
> Delete an unreachable peer will triggers all regionservers abort
>
>
> Key: HBASE-19925
> URL: https://issues.apache.org/jira/browse/HBASE-19925
> Project: HBase
> Issue Type: Bug
>Reporter: Yun Zhao
>Assignee: Yun Zhao
>Priority: Critical
> Attachments: HBASE-19925.master.001.patch
>
>
> Add an unreachable peer
> {code:java}
> add_peer '4', CLUSTER_KEY => "server1.cie.com:2181:/hbase"{code}
> After a while to delete it,Regionserver will appear in the following log and
> stop.
> {code:java}
> 2018-02-02 20:04:25,959 INFO [main-EventThread.replicationSource,4]
> regionserver.ReplicationSource: Replicating
> 5467de52-dc46-45be-902c-110dd7a83e06 -> null
> 2018-02-02 20:04:25,960 ERROR
> [main-EventThread.replicationSource,4.replicationSource..com%2C16020%2C1515498473547.default,4]
> regionserver.ReplicationSource: Unexpected exception in
> ReplicationSourceWorkerThread, currentPath=null
> java.lang.IllegalArgumentException: Peer with id= 4 is not connected
> at
> org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.getStatusOfPeer(ReplicationPeersZKImpl.java:207)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.isPeerEnabled(ReplicationSource.java:327)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceWorkerThread.run(ReplicationSource.java:512)
> 2018-02-02 20:04:25,960 INFO
> [main-EventThread.replicationSource,4.replicationSource..com%2C16020%2C1515498473547.default,4]
> regionserver.HRegionServer: STOPPED: Unexpected exception in
> ReplicationSourceWorkerThread{code}
>
> HBase 1.2.6
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19917) Improve RSGroupBasedLoadBalancer#filterServers() to be more efficient
[
https://issues.apache.org/jira/browse/HBASE-19917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351389#comment-16351389
]
Ted Yu commented on HBASE-19917:
{code}
298 } else { // not a HashSet yet
299 serverAddressSet = new HashSet(servers);
{code}
Can you identify the caller(s) where non-HashSet parameter is passed ?
If possible, we should change those to using HashSet.
> Improve RSGroupBasedLoadBalancer#filterServers() to be more efficient
> -
>
> Key: HBASE-19917
> URL: https://issues.apache.org/jira/browse/HBASE-19917
> Project: HBase
> Issue Type: Improvement
> Components: rsgroup
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Attachments: HBASE-19917.master.000.patch
>
>
> {code:title=hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java|borderStyle=solid}
> private List filterServers(Collection servers,
> Collection onlineServers) {
> ArrayList finalList = new ArrayList();
> for (Address server : servers) {
> for(ServerName curr: onlineServers) {
> if(curr.getAddress().equals(server)) {
> finalList.add(curr);
> }
> }
> }
> return finalList;
> }
> {code}
> filterServers is to return the union of servers and onlineServers. The
> current implementation has time complexity as O(m * n) (2 loops), could be in
> O(m + n) if HashSet is used. The trade-off is space complexity is increased.
> Another point which could be improved: filterServers() is only called in
> filterOfflineServers(). filterOfflineServers calls filterServers(Set, List).
> The current filterServers(Collection, Collection) seems could be improved.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19925) Delete an unreachable peer will triggers all regionservers abort
[
https://issues.apache.org/jira/browse/HBASE-19925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351392#comment-16351392
]
Duo Zhang commented on HBASE-19925:
---
And mind adding a test for this? Now we only test the scenario where we even do
not have the correct zk so it will be stuck in the first loop for creating
ReplicationEndpoint.
Oh, the first loop have the same problem... It will just exit the loop if
source is inactive... Maybe we should also change it to a 'for(;;)' loop, and
if we find that the source is inactive, we just return, instead of break.
Thanks.
> Delete an unreachable peer will triggers all regionservers abort
>
>
> Key: HBASE-19925
> URL: https://issues.apache.org/jira/browse/HBASE-19925
> Project: HBase
> Issue Type: Bug
>Reporter: Yun Zhao
>Assignee: Yun Zhao
>Priority: Critical
> Attachments: HBASE-19925.master.001.patch
>
>
> Add an unreachable peer
> {code:java}
> add_peer '4', CLUSTER_KEY => "server1.cie.com:2181:/hbase"{code}
> After a while to delete it,Regionserver will appear in the following log and
> stop.
> {code:java}
> 2018-02-02 20:04:25,959 INFO [main-EventThread.replicationSource,4]
> regionserver.ReplicationSource: Replicating
> 5467de52-dc46-45be-902c-110dd7a83e06 -> null
> 2018-02-02 20:04:25,960 ERROR
> [main-EventThread.replicationSource,4.replicationSource..com%2C16020%2C1515498473547.default,4]
> regionserver.ReplicationSource: Unexpected exception in
> ReplicationSourceWorkerThread, currentPath=null
> java.lang.IllegalArgumentException: Peer with id= 4 is not connected
> at
> org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.getStatusOfPeer(ReplicationPeersZKImpl.java:207)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.isPeerEnabled(ReplicationSource.java:327)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource$ReplicationSourceWorkerThread.run(ReplicationSource.java:512)
> 2018-02-02 20:04:25,960 INFO
> [main-EventThread.replicationSource,4.replicationSource..com%2C16020%2C1515498473547.default,4]
> regionserver.HRegionServer: STOPPED: Unexpected exception in
> ReplicationSourceWorkerThread{code}
>
> HBase 1.2.6
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19926) Use a separated class to implement the WALActionListener for Replication
[ https://issues.apache.org/jira/browse/HBASE-19926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351394#comment-16351394 ] Duo Zhang commented on HBASE-19926: --- [~appy] FYI. > Use a separated class to implement the WALActionListener for Replication > > > Key: HBASE-19926 > URL: https://issues.apache.org/jira/browse/HBASE-19926 > Project: HBase > Issue Type: Bug > Components: Replication, wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19926.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19926) Use a separated class to implement the WALActionListener for Replication
[ https://issues.apache.org/jira/browse/HBASE-19926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19926: -- Attachment: HBASE-19926.patch > Use a separated class to implement the WALActionListener for Replication > > > Key: HBASE-19926 > URL: https://issues.apache.org/jira/browse/HBASE-19926 > Project: HBase > Issue Type: Bug > Components: Replication, wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19926.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19926) Use a separated class to implement the WALActionListener for Replication
[ https://issues.apache.org/jira/browse/HBASE-19926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19926: -- Status: Patch Available (was: Open) > Use a separated class to implement the WALActionListener for Replication > > > Key: HBASE-19926 > URL: https://issues.apache.org/jira/browse/HBASE-19926 > Project: HBase > Issue Type: Bug > Components: Replication, wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19926.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19900) Region-level exception destroy the result of batch
[
https://issues.apache.org/jira/browse/HBASE-19900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351404#comment-16351404
]
Hadoop QA commented on HBASE-19900:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
9s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} hbase-client: The patch generated 0 new + 24
unchanged - 4 fixed = 24 total (was 28) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} The patch hbase-server passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
33s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
17m 58s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 49s{color}
| {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}119m 20s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}161m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestAsyncProcess |
| | hadoop.hbase.client.TestAsyncTableGetMultiThreaded |
| | hadoop.hbase.client.TestTableSnapshotScanner |
| | hadoop.hbase.client.TestFromClientSide3 |
| | hadoop.hbase.regionserver.TestEndToEndSplitTransaction |
| | hadoop.hbase.client.TestMultiParallel |
| | hadoop.hbase.master.procedure.TestTruncateTableProcedure |
| | hadoop.hbase.regionserver.wal.TestWALReplay |
| | hadoop.hbase.client.TestMultiRespectsLimits |
| | hadoop.hbase.client.TestAdmin2 |
| | hadoop.hbase.util.TestFromClientSide3WoUnsafe |
| | hadoop.hbase.regionserver.TestCompactSplitThread |
\\
\\
|| Subsystem || Rep
[jira] [Commented] (HBASE-19920) TokenUtil.obtainToken unnecessarily creates a local directory
[
https://issues.apache.org/jira/browse/HBASE-19920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351414#comment-16351414
]
Mike Drob commented on HBASE-19920:
---
[~rohini] - what version of HBase were you using, so that we know to set the
fixversion appropriately
> TokenUtil.obtainToken unnecessarily creates a local directory
> -
>
> Key: HBASE-19920
> URL: https://issues.apache.org/jira/browse/HBASE-19920
> Project: HBase
> Issue Type: Bug
>Reporter: Rohini Palaniswamy
>Assignee: Mike Drob
>Priority: Major
> Fix For: 2.0
>
> Attachments: HBASE-19920.patch
>
>
> On client code, when one calls TokenUtil.obtainToken it loads ProtobufUtil
> which in its static block initializes DynamicClassLoader and that creates the
> directory ${hbase.local.dir}/jars/ and also instantiates a filesystem class
> to access hbase.dynamic.jars.dir.
> https://github.com/apache/hbase/blob/master/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DynamicClassLoader.java#L109-L127
> Since this is region server specific code, not expecting this to happen when
> one accesses hbase as a client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19900) Region-level exception destroy the result of batch
[ https://issues.apache.org/jira/browse/HBASE-19900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351416#comment-16351416 ] Chia-Ping Tsai commented on HBASE-19900: Seems some tests expect that failed ops in RetriesExhaustedWithDetailsException will have the different exceptions. > Region-level exception destroy the result of batch > -- > > Key: HBASE-19900 > URL: https://issues.apache.org/jira/browse/HBASE-19900 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Critical > Fix For: 1.3.2, 1.5.0, 1.2.7, 2.0.0-beta-2, 1.4.2 > > Attachments: HBASE-19900.v0.patch > > > 1) decrease action count repeatedly > If the AsyncRequestFuture#waitUntilDone return prematurely, user will get the > incorrect results. Or user will be block by AsyncRequestFuture#waitUntilDone > as the count is never equal with 0. > 2) the successive result will be overwrited > 3) The failed op is added to RetriesExhaustedWithDetailsException repeatedly > AsyncRequestFutureImpl#receiveMultiAction process the action-lever error > first, and then add the region-level exception to each action. Hence, user > may get the various exceptions for the same action (row op) from the > RetriesExhaustedWithDetailsException. > In fact, if both of action-level exception and region-lever exception exist, > they always have the same context. I'm not sure whether that is what > RetriesExhaustedWithDetailsException want. As i see it, we shouldn't have the > duplicate ops in RetriesExhaustedWithDetailsException since that may confuse > users if they catch the RetriesExhaustedWithDetailsException to check the > invalid operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19926) Use a separated class to implement the WALActionListener for Replication
[
https://issues.apache.org/jira/browse/HBASE-19926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351417#comment-16351417
]
Hadoop QA commented on HBASE-19926:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
55s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
49s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
4s{color} | {color:red} hbase-server: The patch generated 2 new + 14 unchanged
- 0 fixed = 16 total (was 14) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
47s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
18m 26s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}100m
45s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}139m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19926 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12909098/HBASE-19926.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux c42fcb0697a8 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 6519b98ac3 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/11376/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/11376/testReport/ |
| Max. process+thread count | 4860 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/11376/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message wa
[jira] [Commented] (HBASE-19920) TokenUtil.obtainToken unnecessarily creates a local directory
[
https://issues.apache.org/jira/browse/HBASE-19920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351423#comment-16351423
]
Attila Sasvari commented on HBASE-19920:
[~mdrob] we were using 1.2.0 (cdh5-1.2.0_5.13).
> TokenUtil.obtainToken unnecessarily creates a local directory
> -
>
> Key: HBASE-19920
> URL: https://issues.apache.org/jira/browse/HBASE-19920
> Project: HBase
> Issue Type: Bug
>Reporter: Rohini Palaniswamy
>Assignee: Mike Drob
>Priority: Major
> Fix For: 2.0
>
> Attachments: HBASE-19920.patch
>
>
> On client code, when one calls TokenUtil.obtainToken it loads ProtobufUtil
> which in its static block initializes DynamicClassLoader and that creates the
> directory ${hbase.local.dir}/jars/ and also instantiates a filesystem class
> to access hbase.dynamic.jars.dir.
> https://github.com/apache/hbase/blob/master/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DynamicClassLoader.java#L109-L127
> Since this is region server specific code, not expecting this to happen when
> one accesses hbase as a client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19920) TokenUtil.obtainToken unnecessarily creates a local directory
[
https://issues.apache.org/jira/browse/HBASE-19920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351430#comment-16351430
]
Ted Yu commented on HBASE-19920:
{code}
34 * @return Exception wrapped in ServiceException or
35 * a new IOException that wraps the unexpected ServiceException.
36 */
37public static IOException
getRemoteException(com.google.protobuf.ServiceException se) {
{code}
For @return, IOException is returned.
{code}
61 * @param e
62 */
63public static IOException handleRemoteException(Exception e) {
{code}
Add comment for @param
Are you going to add a test case in next patch ?
Please check unit test failure.
> TokenUtil.obtainToken unnecessarily creates a local directory
> -
>
> Key: HBASE-19920
> URL: https://issues.apache.org/jira/browse/HBASE-19920
> Project: HBase
> Issue Type: Bug
>Reporter: Rohini Palaniswamy
>Assignee: Mike Drob
>Priority: Major
> Fix For: 2.0
>
> Attachments: HBASE-19920.patch
>
>
> On client code, when one calls TokenUtil.obtainToken it loads ProtobufUtil
> which in its static block initializes DynamicClassLoader and that creates the
> directory ${hbase.local.dir}/jars/ and also instantiates a filesystem class
> to access hbase.dynamic.jars.dir.
> https://github.com/apache/hbase/blob/master/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DynamicClassLoader.java#L109-L127
> Since this is region server specific code, not expecting this to happen when
> one accesses hbase as a client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19919) Tidying up logging
[ https://issues.apache.org/jira/browse/HBASE-19919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351437#comment-16351437 ] Hudson commented on HBASE-19919: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4519 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4519/]) Revert "HBASE-19919 Tidying up logging" (zhangduo: rev 6519b98ac3115c4442a2778f6ed7b39ce5cd3b83) * (delete) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java.rej * (delete) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationPeerConfigUpgrader.java.rej * (delete) hbase-common/src/test/java/org/apache/hadoop/hbase/net/TestAddress.java > Tidying up logging > -- > > Key: HBASE-19919 > URL: https://issues.apache.org/jira/browse/HBASE-19919 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: 0001-HBASE-19919-Tidying-up-logging.patch, > HBASE-19919.branch-2.001.patch > > > Reading logs, there is a bunch of stuff we don't need, thread names are too > long, etc. Doing a little tidying. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19919) Tidying up logging
[ https://issues.apache.org/jira/browse/HBASE-19919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351443#comment-16351443 ] stack commented on HBASE-19919: --- Ugh. Thanks [~Apache9]. I messed up the cherry-pick. Thanks for the revert. Let me fix. > Tidying up logging > -- > > Key: HBASE-19919 > URL: https://issues.apache.org/jira/browse/HBASE-19919 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: 0001-HBASE-19919-Tidying-up-logging.patch, > HBASE-19919.branch-2.001.patch > > > Reading logs, there is a bunch of stuff we don't need, thread names are too > long, etc. Doing a little tidying. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19904) Break dependency of WAL constructor on Replication
[ https://issues.apache.org/jira/browse/HBASE-19904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351444#comment-16351444 ] stack commented on HBASE-19904: --- Did this make it into master branch? > Break dependency of WAL constructor on Replication > -- > > Key: HBASE-19904 > URL: https://issues.apache.org/jira/browse/HBASE-19904 > Project: HBase > Issue Type: Improvement > Components: Replication, wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19904-branch-2.patch, HBASE-19904-branch-2.patch, > HBASE-19904-v3.patch, HBASE-19904-v3.patch, HBASE-19904-v4.patch, > HBASE-19904-v4.patch, HBASE-19904-v5.patch > > > When implementing synchronous replication, I found that we need to depend > more on replication in WAL so it is even more pain... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19528) Major Compaction Tool
[ https://issues.apache.org/jira/browse/HBASE-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351446#comment-16351446 ] stack commented on HBASE-19528: --- [~churromorales] What you think of the hadoop build failures? [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile (default-compile) on project hbase-server: Compilation failure: Compilation failure: [ERROR] /testptch/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java:[49,39] cannot find symbol [ERROR] symbol: class BlockStoragePolicy [ERROR] location: package org.apache.hadoop.hdfs.protocol [ERROR] /testptch/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java:[55,53] cannot find symbol [ERROR] symbol: class BlockStoragePolicySuite [ERROR] location: package org.apache.hadoop.hdfs.server.blockmanagement [ERROR] /testptch/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java:[477,35] cannot find symbol [ERROR] symbol: class BlockStoragePolicySuite [ERROR] location: class org.apache.hadoop.hbase.fs.HFileSystem [ERROR] /testptch/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java:[478,64] cannot find symbol [ERROR] symbol: class BlockStoragePolicySuite [ERROR] location: class org.apache.hadoop.hbase.fs.HFileSystem [ERROR] /testptch/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java:[480,40] cannot find symbol [ERROR] symbol: method getStoragePolicy() [ERROR] location: variable status of type org.apache.hadoop.hdfs.protocol.HdfsFileStatus [ERROR] /testptch/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java:[482,13] cannot find symbol [ERROR] symbol: class BlockStoragePolicy [ERROR] location: class org.apache.hadoop.hbase.fs.HFileSystem [ERROR] /testptch/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java:[482,48] cannot find symbol [ERROR] symbol: method getStoragePolicies() [ERROR] location: variable dfs of type org.apache.hadoop.hdfs.DistributedFileSystem [ERROR] /testptch/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java:[483,18] cannot find symbol [ERROR] symbol: class BlockStoragePolicy [ERROR] location: class org.apache.hadoop.hbase.fs.HFileSystem [ERROR] -> [Help 1] [ERROR] Using API that is not in the old Hadoops? Thanks. > Major Compaction Tool > -- > > Key: HBASE-19528 > URL: https://issues.apache.org/jira/browse/HBASE-19528 > Project: HBase > Issue Type: New Feature >Reporter: churro morales >Assignee: churro morales >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: 0001-HBASE-19528-Major-Compaction-Tool-ADDENDUM.patch, > HBASE-19528.branch-1.patch, HBASE-19528.patch, HBASE-19528.v1.branch-1.patch, > HBASE-19528.v1.patch, HBASE-19528.v2.branch-1.patch, > HBASE-19528.v2.branch-1.patch, HBASE-19528.v2.branch-1.patch, > HBASE-19528.v2.branch-1.patch, HBASE-19528.v8.patch > > > The basic overview of how this tool works is: > Parameters: > Table > Stores > ClusterConcurrency > Timestamp > So you input a table, desired concurrency and the list of stores you wish to > major compact. The tool first checks the filesystem to see which stores need > compaction based on the timestamp you provide (default is current time). It > takes that list of stores that require compaction and executes those requests > concurrently with at most N distinct RegionServers compacting at a given > time. Each thread waits for the compaction to complete before moving to the > next queue. If a region split, merge or move happens this tool ensures those > regions get major compacted as well. > This helps us in two ways, we can limit how much I/O bandwidth we are using > for major compaction cluster wide and we are guaranteed after the tool > completes that all requested compactions complete regardless of moves, merges > and splits. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19919) Tidying up logging
[ https://issues.apache.org/jira/browse/HBASE-19919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351449#comment-16351449 ] stack commented on HBASE-19919: --- Pushed to master again. > Tidying up logging > -- > > Key: HBASE-19919 > URL: https://issues.apache.org/jira/browse/HBASE-19919 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: 0001-HBASE-19919-Tidying-up-logging.patch, > HBASE-19919.branch-2.001.patch > > > Reading logs, there is a bunch of stuff we don't need, thread names are too > long, etc. Doing a little tidying. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19904) Break dependency of WAL constructor on Replication
[ https://issues.apache.org/jira/browse/HBASE-19904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351450#comment-16351450 ] stack commented on HBASE-19904: --- nvm. I see it now. Ignore above. > Break dependency of WAL constructor on Replication > -- > > Key: HBASE-19904 > URL: https://issues.apache.org/jira/browse/HBASE-19904 > Project: HBase > Issue Type: Improvement > Components: Replication, wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19904-branch-2.patch, HBASE-19904-branch-2.patch, > HBASE-19904-v3.patch, HBASE-19904-v3.patch, HBASE-19904-v4.patch, > HBASE-19904-v4.patch, HBASE-19904-v5.patch > > > When implementing synchronous replication, I found that we need to depend > more on replication in WAL so it is even more pain... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (HBASE-19904) Break dependency of WAL constructor on Replication
[ https://issues.apache.org/jira/browse/HBASE-19904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19904: -- Comment: was deleted (was: nvm. I see it now. Ignore above.) > Break dependency of WAL constructor on Replication > -- > > Key: HBASE-19904 > URL: https://issues.apache.org/jira/browse/HBASE-19904 > Project: HBase > Issue Type: Improvement > Components: Replication, wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19904-branch-2.patch, HBASE-19904-branch-2.patch, > HBASE-19904-v3.patch, HBASE-19904-v3.patch, HBASE-19904-v4.patch, > HBASE-19904-v4.patch, HBASE-19904-v5.patch > > > When implementing synchronous replication, I found that we need to depend > more on replication in WAL so it is even more pain... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (HBASE-19904) Break dependency of WAL constructor on Replication
[ https://issues.apache.org/jira/browse/HBASE-19904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19904: -- Comment: was deleted (was: Did this make it into master branch?) > Break dependency of WAL constructor on Replication > -- > > Key: HBASE-19904 > URL: https://issues.apache.org/jira/browse/HBASE-19904 > Project: HBase > Issue Type: Improvement > Components: Replication, wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19904-branch-2.patch, HBASE-19904-branch-2.patch, > HBASE-19904-v3.patch, HBASE-19904-v3.patch, HBASE-19904-v4.patch, > HBASE-19904-v4.patch, HBASE-19904-v5.patch > > > When implementing synchronous replication, I found that we need to depend > more on replication in WAL so it is even more pain... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19553) Old replica regions should be cleared from AM memory after primary region split or merge
[ https://issues.apache.org/jira/browse/HBASE-19553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351453#comment-16351453 ] huaxiang sun commented on HBASE-19553: -- Sorry for the late response, I will take a look, thanks. > Old replica regions should be cleared from AM memory after primary region > split or merge > > > Key: HBASE-19553 > URL: https://issues.apache.org/jira/browse/HBASE-19553 > Project: HBase > Issue Type: Bug > Components: master >Reporter: huaxiang sun >Assignee: Pankaj Kumar >Priority: Minor > Fix For: 1.5.0 > > Attachments: HBASE-19553-branch-1-v2.patch, > HBASE-19553-branch-1-v3.patch, HBASE-19553-branch-1-v4.patch, > HBASE-19553-branch-1.patch > > > Similar to HBASE-18025, the replica parent's info is not removed from master. > Actually I think it can be removed after replica region is split or merged, I > will check the logic and apply one patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19553) Old replica regions should be cleared from AM memory after primary region split or merge
[ https://issues.apache.org/jira/browse/HBASE-19553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19553: -- Attachment: HBASE-19553-branch-1-v4.patch > Old replica regions should be cleared from AM memory after primary region > split or merge > > > Key: HBASE-19553 > URL: https://issues.apache.org/jira/browse/HBASE-19553 > Project: HBase > Issue Type: Bug > Components: master >Reporter: huaxiang sun >Assignee: Pankaj Kumar >Priority: Minor > Fix For: 1.5.0 > > Attachments: HBASE-19553-branch-1-v2.patch, > HBASE-19553-branch-1-v3.patch, HBASE-19553-branch-1-v4.patch, > HBASE-19553-branch-1-v4.patch, HBASE-19553-branch-1.patch > > > Similar to HBASE-18025, the replica parent's info is not removed from master. > Actually I think it can be removed after replica region is split or merged, I > will check the logic and apply one patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19528) Major Compaction Tool
[ https://issues.apache.org/jira/browse/HBASE-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351455#comment-16351455 ] churro morales commented on HBASE-19528: Yes you are absolutely correct, I'll fix thanks > Major Compaction Tool > -- > > Key: HBASE-19528 > URL: https://issues.apache.org/jira/browse/HBASE-19528 > Project: HBase > Issue Type: New Feature >Reporter: churro morales >Assignee: churro morales >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: 0001-HBASE-19528-Major-Compaction-Tool-ADDENDUM.patch, > HBASE-19528.branch-1.patch, HBASE-19528.patch, HBASE-19528.v1.branch-1.patch, > HBASE-19528.v1.patch, HBASE-19528.v2.branch-1.patch, > HBASE-19528.v2.branch-1.patch, HBASE-19528.v2.branch-1.patch, > HBASE-19528.v2.branch-1.patch, HBASE-19528.v8.patch > > > The basic overview of how this tool works is: > Parameters: > Table > Stores > ClusterConcurrency > Timestamp > So you input a table, desired concurrency and the list of stores you wish to > major compact. The tool first checks the filesystem to see which stores need > compaction based on the timestamp you provide (default is current time). It > takes that list of stores that require compaction and executes those requests > concurrently with at most N distinct RegionServers compacting at a given > time. Each thread waits for the compaction to complete before moving to the > next queue. If a region split, merge or move happens this tool ensures those > regions get major compacted as well. > This helps us in two ways, we can limit how much I/O bandwidth we are using > for major compaction cluster wide and we are guaranteed after the tool > completes that all requested compactions complete regardless of moves, merges > and splits. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19919) Tidying up logging
[ https://issues.apache.org/jira/browse/HBASE-19919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351456#comment-16351456 ] stack commented on HBASE-19919: --- And had to add an addendum subsequent to the master fixup. So, push to branch-2 followed by an addendum and a push to master that was broke, a revert, a repush, and then an addendum. Pardon my mess! > Tidying up logging > -- > > Key: HBASE-19919 > URL: https://issues.apache.org/jira/browse/HBASE-19919 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: 0001-HBASE-19919-Tidying-up-logging.patch, > HBASE-19919.branch-2.001.patch > > > Reading logs, there is a bunch of stuff we don't need, thread names are too > long, etc. Doing a little tidying. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-19919) Tidying up logging
[ https://issues.apache.org/jira/browse/HBASE-19919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-19919. --- Resolution: Fixed Reresolving > Tidying up logging > -- > > Key: HBASE-19919 > URL: https://issues.apache.org/jira/browse/HBASE-19919 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: 0001-HBASE-19919-Tidying-up-logging.patch, > HBASE-19919.branch-2.001.patch > > > Reading logs, there is a bunch of stuff we don't need, thread names are too > long, etc. Doing a little tidying. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19921) Disable 1.1 nightly builds.
[ https://issues.apache.org/jira/browse/HBASE-19921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351459#comment-16351459 ] stack commented on HBASE-19921: --- bq. Would it have been better to delete the whole branch? Considered that but thought it premature... Perhaps we should. > Disable 1.1 nightly builds. > --- > > Key: HBASE-19921 > URL: https://issues.apache.org/jira/browse/HBASE-19921 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 1.1.13 > > Attachments: 0001-HBASE-19921-Disable-1.1-nightly-builds.patch > > > As suggested by [~mdrob], remove the JenkinsFile is the trick (Nightlies run > for all branches in hbase). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-19927) TestFullLogReconstruction flakey
stack created HBASE-19927: - Summary: TestFullLogReconstruction flakey Key: HBASE-19927 URL: https://issues.apache.org/jira/browse/HBASE-19927 Project: HBase Issue Type: Sub-task Components: wal Reporter: stack Fix For: 2.0.0-beta-2 Fails pretty frequently in hadoopqa builds. There is a recent hang in org.apache.hadoop.hbase.TestFullLogReconstruction.tearDownAfterClass(TestFullLogReconstruction.java:68) In here... https://builds.apache.org/job/PreCommit-HBASE-Build/11363/testReport/org.apache.hadoop.hbase/TestFullLogReconstruction/org_apache_hadoop_hbase_TestFullLogReconstruction/ ... see here. Thread 1250 (RS_CLOSE_META-edd281aedb18:59863-0): State: TIMED_WAITING Blocked count: 92 Waited count: 278 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeFlushMarker(WALUtil.java:81) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) We missed a signal? We need to do an interrupt? The log is not all there in hadoopqa builds so hard to see all that is going on. This test is not in the flakey set either -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19919) Tidying up logging
[ https://issues.apache.org/jira/browse/HBASE-19919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351470#comment-16351470 ] Hudson commented on HBASE-19919: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4520 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4520/]) HBASE-19919 Tidying up logging (stack: rev 06dec205826a6e96e2286180460e5fe014b46fc8) * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/ChoreService.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ChunkCreator.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/StateMachineProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcConnection.java * (add) hbase-common/src/test/java/org/apache/hadoop/hbase/net/TestAddress.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java * (edit) hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ZKProcedureCoordinator.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationPeerConfigUpgrader.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZKSplitLogManagerCoordination.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationLogCleaner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/net/Address.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ZKProcedureUtil.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ReadOnlyZKClient.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java > Tidying up logging > -- > > Key: HBASE-19919 > URL: https://issues.apache.org/jira/browse/HBASE-19919 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: 0001-HBASE-19919-Tidying-up-logging.patch, > HBASE-19919.branch-2.001.patch > > > Reading logs, there is a bunch of stuff we don't need, thread names are too > long, etc. Doing a little tidying. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-19928) TestVisibilityLabelsOnNewVersionBehaviorTable fails
stack created HBASE-19928: - Summary: TestVisibilityLabelsOnNewVersionBehaviorTable fails Key: HBASE-19928 URL: https://issues.apache.org/jira/browse/HBASE-19928 Project: HBase Issue Type: Sub-task Components: test Reporter: stack Assignee: stack Failing 100% of the time w/ below Simple fix. Failed org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable.org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable Failing for the past 43 builds (Since Unstable#1196 ) Took 1.2 sec. add description Error Message The HBaseClassTestRule ClassRule in org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable is for org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes expected: but was: Stacktrace java.lang.AssertionError: The HBaseClassTestRule ClassRule in org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable is for org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes expected: but was: -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19928) TestVisibilityLabelsOnNewVersionBehaviorTable fails
[ https://issues.apache.org/jira/browse/HBASE-19928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19928: -- Attachment: 0001-HBASE-19928-TestVisibilityLabelsOnNewVersionBehavior.patch > TestVisibilityLabelsOnNewVersionBehaviorTable fails > --- > > Key: HBASE-19928 > URL: https://issues.apache.org/jira/browse/HBASE-19928 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: > 0001-HBASE-19928-TestVisibilityLabelsOnNewVersionBehavior.patch > > > Failing 100% of the time w/ below Simple fix. > Failed > org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable.org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable > Failing for the past 43 builds (Since Unstable#1196 ) > Took 1.2 sec. > add description > Error Message > The HBaseClassTestRule ClassRule in > org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable > is for > org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes > expected: org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable> > but was: org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes> > Stacktrace > java.lang.AssertionError: The HBaseClassTestRule ClassRule in > org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable > is for > org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes > expected: org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable> > but was: org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes> -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-19928) TestVisibilityLabelsOnNewVersionBehaviorTable fails
[ https://issues.apache.org/jira/browse/HBASE-19928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-19928. --- Resolution: Fixed Fix Version/s: 2.0.0-beta-2 Pushed the attached to master and branch-2. > TestVisibilityLabelsOnNewVersionBehaviorTable fails > --- > > Key: HBASE-19928 > URL: https://issues.apache.org/jira/browse/HBASE-19928 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: > 0001-HBASE-19928-TestVisibilityLabelsOnNewVersionBehavior.patch > > > Failing 100% of the time w/ below Simple fix. > Failed > org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable.org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable > Failing for the past 43 builds (Since Unstable#1196 ) > Took 1.2 sec. > add description > Error Message > The HBaseClassTestRule ClassRule in > org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable > is for > org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes > expected: org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable> > but was: org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes> > Stacktrace > java.lang.AssertionError: The HBaseClassTestRule ClassRule in > org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable > is for > org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes > expected: org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable> > but was: org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes> -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19553) Old replica regions should be cleared from AM memory after primary region split or merge
[
https://issues.apache.org/jira/browse/HBASE-19553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351506#comment-16351506
]
Hadoop QA commented on HBASE-19553:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} branch-1 passed with JDK v1.8.0_162 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} branch-1 passed with JDK v1.7.0_171 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
52s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} branch-1 passed with JDK v1.8.0_162 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} branch-1 passed with JDK v1.7.0_171 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed with JDK v1.8.0_162 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed with JDK v1.7.0_171 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
32s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m
24s{color} | {color:red} The patch causes 44 errors with Hadoop v2.4.1. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m
16s{color} | {color:red} The patch causes 44 errors with Hadoop v2.5.2. {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed with JDK v1.8.0_162 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed with JDK v1.7.0_171 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 12s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}133m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.TestCatalogJanitorInMemoryStates |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:36a7029 |
| JIRA Issue | HBASE-19553 |
| JIRA Patch URL |
https://issues.apach
[jira] [Commented] (HBASE-19919) Tidying up logging
[ https://issues.apache.org/jira/browse/HBASE-19919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351578#comment-16351578 ] Hudson commented on HBASE-19919: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4521 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4521/]) HBASE-19919 Tidying up logging; ADDENDUM Fix tests w/ mocked Servers (stack: rev 12f3c82a866eee0436c22136909882581fd19905) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java > Tidying up logging > -- > > Key: HBASE-19919 > URL: https://issues.apache.org/jira/browse/HBASE-19919 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: 0001-HBASE-19919-Tidying-up-logging.patch, > HBASE-19919.branch-2.001.patch > > > Reading logs, there is a bunch of stuff we don't need, thread names are too > long, etc. Doing a little tidying. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19928) TestVisibilityLabelsOnNewVersionBehaviorTable fails
[ https://issues.apache.org/jira/browse/HBASE-19928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351579#comment-16351579 ] Hudson commented on HBASE-19928: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4521 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4521/]) HBASE-19928 TestVisibilityLabelsOnNewVersionBehaviorTable fails (stack: rev bfd74686c75e8481e4662bbdc519bf4f98e9600c) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsOnNewVersionBehaviorTable.java > TestVisibilityLabelsOnNewVersionBehaviorTable fails > --- > > Key: HBASE-19928 > URL: https://issues.apache.org/jira/browse/HBASE-19928 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: > 0001-HBASE-19928-TestVisibilityLabelsOnNewVersionBehavior.patch > > > Failing 100% of the time w/ below Simple fix. > Failed > org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable.org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable > Failing for the past 43 builds (Since Unstable#1196 ) > Took 1.2 sec. > add description > Error Message > The HBaseClassTestRule ClassRule in > org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable > is for > org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes > expected: org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable> > but was: org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes> > Stacktrace > java.lang.AssertionError: The HBaseClassTestRule ClassRule in > org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable > is for > org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes > expected: org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable> > but was: org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes> -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19927) TestFullLogReconstruction flakey
[ https://issues.apache.org/jira/browse/HBASE-19927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351581#comment-16351581 ] Duo Zhang commented on HBASE-19927: --- IIRC this happened before. I used to introduce a close chore in AsyncFSWAL to solve it and later removed it. Let me revisit the commit history to find out what happened. > TestFullLogReconstruction flakey > > > Key: HBASE-19927 > URL: https://issues.apache.org/jira/browse/HBASE-19927 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: stack >Priority: Major > Fix For: 2.0.0-beta-2 > > > Fails pretty frequently in hadoopqa builds. > There is a recent hang in > org.apache.hadoop.hbase.TestFullLogReconstruction.tearDownAfterClass(TestFullLogReconstruction.java:68) > In here... > https://builds.apache.org/job/PreCommit-HBASE-Build/11363/testReport/org.apache.hadoop.hbase/TestFullLogReconstruction/org_apache_hadoop_hbase_TestFullLogReconstruction/ > ... see here. > Thread 1250 (RS_CLOSE_META-edd281aedb18:59863-0): > State: TIMED_WAITING > Blocked count: 92 > Waited count: 278 > Stack: > java.lang.Object.wait(Native Method) > > org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) > > org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) > > org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) > > org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) > > org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeFlushMarker(WALUtil.java:81) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) > org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) > org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) > > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748) > We missed a signal? We need to do an interrupt? The log is not all there in > hadoopqa builds so hard to see all that is going on. This test is not in the > flakey set either -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19928) TestVisibilityLabelsOnNewVersionBehaviorTable fails
[ https://issues.apache.org/jira/browse/HBASE-19928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351582#comment-16351582 ] Duo Zhang commented on HBASE-19928: --- HBASE-19914 is for it... Anyway, let me rebase the patch there and try again... > TestVisibilityLabelsOnNewVersionBehaviorTable fails > --- > > Key: HBASE-19928 > URL: https://issues.apache.org/jira/browse/HBASE-19928 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: > 0001-HBASE-19928-TestVisibilityLabelsOnNewVersionBehavior.patch > > > Failing 100% of the time w/ below Simple fix. > Failed > org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable.org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable > Failing for the past 43 builds (Since Unstable#1196 ) > Took 1.2 sec. > add description > Error Message > The HBaseClassTestRule ClassRule in > org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable > is for > org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes > expected: org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable> > but was: org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes> > Stacktrace > java.lang.AssertionError: The HBaseClassTestRule ClassRule in > org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable > is for > org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes > expected: org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsOnNewVersionBehaviorTable> > but was: org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes> -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19926) Use a separated class to implement the WALActionListener for Replication
[ https://issues.apache.org/jira/browse/HBASE-19926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351583#comment-16351583 ] Duo Zhang commented on HBASE-19926: --- Any concerns? [~appy]. Thanks. > Use a separated class to implement the WALActionListener for Replication > > > Key: HBASE-19926 > URL: https://issues.apache.org/jira/browse/HBASE-19926 > Project: HBase > Issue Type: Bug > Components: Replication, wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19926.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19914) Refactor TestVisibilityLabelsOnNewVersionBehaviorTable
[ https://issues.apache.org/jira/browse/HBASE-19914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19914: -- Attachment: HBASE-19914-v2.patch > Refactor TestVisibilityLabelsOnNewVersionBehaviorTable > -- > > Key: HBASE-19914 > URL: https://issues.apache.org/jira/browse/HBASE-19914 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19914-v1.patch, HBASE-19914-v2.patch, > HBASE-19914.patch, HBASE-19914.patch > > > And both TestVisibilityLabelsOnNewVersionBehaviorTable and its parent class > run about 2 minutes, which is not safe to declared as MediumTests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19914) Refactor TestVisibilityLabelsOnNewVersionBehaviorTable
[ https://issues.apache.org/jira/browse/HBASE-19914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351587#comment-16351587 ] Duo Zhang commented on HBASE-19914: --- Rebase. > Refactor TestVisibilityLabelsOnNewVersionBehaviorTable > -- > > Key: HBASE-19914 > URL: https://issues.apache.org/jira/browse/HBASE-19914 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19914-v1.patch, HBASE-19914-v2.patch, > HBASE-19914.patch, HBASE-19914.patch > > > And both TestVisibilityLabelsOnNewVersionBehaviorTable and its parent class > run about 2 minutes, which is not safe to declared as MediumTests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19914) Refactor TestVisibilityLabelsOnNewVersionBehaviorTable
[ https://issues.apache.org/jira/browse/HBASE-19914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19914: -- Summary: Refactor TestVisibilityLabelsOnNewVersionBehaviorTable (was: TestVisibilityLabelsOnNewVersionBehaviorTable does not have Category annotation) > Refactor TestVisibilityLabelsOnNewVersionBehaviorTable > -- > > Key: HBASE-19914 > URL: https://issues.apache.org/jira/browse/HBASE-19914 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19914-v1.patch, HBASE-19914.patch, > HBASE-19914.patch > > > And both TestVisibilityLabelsOnNewVersionBehaviorTable and its parent class > run about 2 minutes, which is not safe to declared as MediumTests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19927) TestFullLogReconstruction flakey
[ https://issues.apache.org/jira/browse/HBASE-19927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351588#comment-16351588 ] Duo Zhang commented on HBASE-19927: --- And I think we should enable TestDLSFSHLog and TestDLSAsyncFSWAL again to fix them also? > TestFullLogReconstruction flakey > > > Key: HBASE-19927 > URL: https://issues.apache.org/jira/browse/HBASE-19927 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: stack >Priority: Major > Fix For: 2.0.0-beta-2 > > > Fails pretty frequently in hadoopqa builds. > There is a recent hang in > org.apache.hadoop.hbase.TestFullLogReconstruction.tearDownAfterClass(TestFullLogReconstruction.java:68) > In here... > https://builds.apache.org/job/PreCommit-HBASE-Build/11363/testReport/org.apache.hadoop.hbase/TestFullLogReconstruction/org_apache_hadoop_hbase_TestFullLogReconstruction/ > ... see here. > Thread 1250 (RS_CLOSE_META-edd281aedb18:59863-0): > State: TIMED_WAITING > Blocked count: 92 > Waited count: 278 > Stack: > java.lang.Object.wait(Native Method) > > org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) > > org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) > > org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) > > org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) > > org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeFlushMarker(WALUtil.java:81) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) > org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) > org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) > > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748) > We missed a signal? We need to do an interrupt? The log is not all there in > hadoopqa builds so hard to see all that is going on. This test is not in the > flakey set either -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19927) TestFullLogReconstruction flakey
[ https://issues.apache.org/jira/browse/HBASE-19927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351590#comment-16351590 ] Duo Zhang commented on HBASE-19927: --- HBASE-17053. Let me check again. > TestFullLogReconstruction flakey > > > Key: HBASE-19927 > URL: https://issues.apache.org/jira/browse/HBASE-19927 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: stack >Priority: Major > Fix For: 2.0.0-beta-2 > > > Fails pretty frequently in hadoopqa builds. > There is a recent hang in > org.apache.hadoop.hbase.TestFullLogReconstruction.tearDownAfterClass(TestFullLogReconstruction.java:68) > In here... > https://builds.apache.org/job/PreCommit-HBASE-Build/11363/testReport/org.apache.hadoop.hbase/TestFullLogReconstruction/org_apache_hadoop_hbase_TestFullLogReconstruction/ > ... see here. > Thread 1250 (RS_CLOSE_META-edd281aedb18:59863-0): > State: TIMED_WAITING > Blocked count: 92 > Waited count: 278 > Stack: > java.lang.Object.wait(Native Method) > > org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) > > org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) > > org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) > > org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) > > org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeFlushMarker(WALUtil.java:81) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) > org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) > org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) > > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748) > We missed a signal? We need to do an interrupt? The log is not all there in > hadoopqa builds so hard to see all that is going on. This test is not in the > flakey set either -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19926) Use a separated class to implement the WALActionListener for Replication
[ https://issues.apache.org/jira/browse/HBASE-19926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19926: -- Attachment: HBASE-19926-v1.patch > Use a separated class to implement the WALActionListener for Replication > > > Key: HBASE-19926 > URL: https://issues.apache.org/jira/browse/HBASE-19926 > Project: HBase > Issue Type: Bug > Components: Replication, wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19926-v1.patch, HBASE-19926.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19926) Use a separated class to implement the WALActionListener for Replication
[ https://issues.apache.org/jira/browse/HBASE-19926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351612#comment-16351612 ] Duo Zhang commented on HBASE-19926: --- Fix checkstyle issues. Will commit if pre commit is OK. > Use a separated class to implement the WALActionListener for Replication > > > Key: HBASE-19926 > URL: https://issues.apache.org/jira/browse/HBASE-19926 > Project: HBase > Issue Type: Bug > Components: Replication, wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19926-v1.patch, HBASE-19926.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19914) Refactor TestVisibilityLabelsOnNewVersionBehaviorTable
[
https://issues.apache.org/jira/browse/HBASE-19914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351616#comment-16351616
]
Hadoop QA commented on HBASE-19914:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
8s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
13s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} The patch hbase-client passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} hbase-server: The patch generated 0 new + 2
unchanged - 13 fixed = 2 total (was 15) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
38s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
18m 15s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
52s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}101m
23s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}144m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19914 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12909119/HBASE-19914-v2.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 6f5923c80f40 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / bfd74686c7 |
| maven | version: Ap
[jira] [Updated] (HBASE-19927) TestFullLogReconstruction flakey
[ https://issues.apache.org/jira/browse/HBASE-19927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19927: -- Attachment: out > TestFullLogReconstruction flakey > > > Key: HBASE-19927 > URL: https://issues.apache.org/jira/browse/HBASE-19927 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: stack >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: js, out > > > Fails pretty frequently in hadoopqa builds. > There is a recent hang in > org.apache.hadoop.hbase.TestFullLogReconstruction.tearDownAfterClass(TestFullLogReconstruction.java:68) > In here... > https://builds.apache.org/job/PreCommit-HBASE-Build/11363/testReport/org.apache.hadoop.hbase/TestFullLogReconstruction/org_apache_hadoop_hbase_TestFullLogReconstruction/ > ... see here. > Thread 1250 (RS_CLOSE_META-edd281aedb18:59863-0): > State: TIMED_WAITING > Blocked count: 92 > Waited count: 278 > Stack: > java.lang.Object.wait(Native Method) > > org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) > > org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) > > org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) > > org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) > > org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeFlushMarker(WALUtil.java:81) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) > org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) > org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) > > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748) > We missed a signal? We need to do an interrupt? The log is not all there in > hadoopqa builds so hard to see all that is going on. This test is not in the > flakey set either -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19927) TestFullLogReconstruction flakey
[ https://issues.apache.org/jira/browse/HBASE-19927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19927: -- Attachment: js > TestFullLogReconstruction flakey > > > Key: HBASE-19927 > URL: https://issues.apache.org/jira/browse/HBASE-19927 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: stack >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: js, out > > > Fails pretty frequently in hadoopqa builds. > There is a recent hang in > org.apache.hadoop.hbase.TestFullLogReconstruction.tearDownAfterClass(TestFullLogReconstruction.java:68) > In here... > https://builds.apache.org/job/PreCommit-HBASE-Build/11363/testReport/org.apache.hadoop.hbase/TestFullLogReconstruction/org_apache_hadoop_hbase_TestFullLogReconstruction/ > ... see here. > Thread 1250 (RS_CLOSE_META-edd281aedb18:59863-0): > State: TIMED_WAITING > Blocked count: 92 > Waited count: 278 > Stack: > java.lang.Object.wait(Native Method) > > org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) > > org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) > > org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) > > org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) > > org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeFlushMarker(WALUtil.java:81) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) > org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) > org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) > > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748) > We missed a signal? We need to do an interrupt? The log is not all there in > hadoopqa builds so hard to see all that is going on. This test is not in the > flakey set either -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19927) TestFullLogReconstruction flakey
[ https://issues.apache.org/jira/browse/HBASE-19927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351624#comment-16351624 ] Duo Zhang commented on HBASE-19927: --- I can reproduce the problem locally, one out of four or five runs. Upload the jstack and test output. Haven't found the root cause yet but obviously the test itself has a problem. In the output you can see that, we shut down the whole cluster before aborting a RS for session expired, so the test just test nothing... First I plan to rewrite the test to make sure that we actually do a 'full log reconstruction'. This could solve the hang also. And then we could try to implement a test to reproduce the problem stably and see how to fix it. Thanks. > TestFullLogReconstruction flakey > > > Key: HBASE-19927 > URL: https://issues.apache.org/jira/browse/HBASE-19927 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: stack >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: js, out > > > Fails pretty frequently in hadoopqa builds. > There is a recent hang in > org.apache.hadoop.hbase.TestFullLogReconstruction.tearDownAfterClass(TestFullLogReconstruction.java:68) > In here... > https://builds.apache.org/job/PreCommit-HBASE-Build/11363/testReport/org.apache.hadoop.hbase/TestFullLogReconstruction/org_apache_hadoop_hbase_TestFullLogReconstruction/ > ... see here. > Thread 1250 (RS_CLOSE_META-edd281aedb18:59863-0): > State: TIMED_WAITING > Blocked count: 92 > Waited count: 278 > Stack: > java.lang.Object.wait(Native Method) > > org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) > > org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) > > org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) > > org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) > > org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeFlushMarker(WALUtil.java:81) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) > org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) > org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) > > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748) > We missed a signal? We need to do an interrupt? The log is not all there in > hadoopqa builds so hard to see all that is going on. This test is not in the > flakey set either -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19927) TestFullLogReconstruction flakey
[ https://issues.apache.org/jira/browse/HBASE-19927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19927: -- Attachment: HBASE-19927.patch > TestFullLogReconstruction flakey > > > Key: HBASE-19927 > URL: https://issues.apache.org/jira/browse/HBASE-19927 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: stack >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19927.patch, js, out > > > Fails pretty frequently in hadoopqa builds. > There is a recent hang in > org.apache.hadoop.hbase.TestFullLogReconstruction.tearDownAfterClass(TestFullLogReconstruction.java:68) > In here... > https://builds.apache.org/job/PreCommit-HBASE-Build/11363/testReport/org.apache.hadoop.hbase/TestFullLogReconstruction/org_apache_hadoop_hbase_TestFullLogReconstruction/ > ... see here. > Thread 1250 (RS_CLOSE_META-edd281aedb18:59863-0): > State: TIMED_WAITING > Blocked count: 92 > Waited count: 278 > Stack: > java.lang.Object.wait(Native Method) > > org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) > > org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) > > org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) > > org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) > > org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeFlushMarker(WALUtil.java:81) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) > org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) > org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) > > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748) > We missed a signal? We need to do an interrupt? The log is not all there in > hadoopqa builds so hard to see all that is going on. This test is not in the > flakey set either -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19927) TestFullLogReconstruction flakey
[ https://issues.apache.org/jira/browse/HBASE-19927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19927: -- Assignee: Duo Zhang Status: Patch Available (was: Open) > TestFullLogReconstruction flakey > > > Key: HBASE-19927 > URL: https://issues.apache.org/jira/browse/HBASE-19927 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: stack >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19927.patch, js, out > > > Fails pretty frequently in hadoopqa builds. > There is a recent hang in > org.apache.hadoop.hbase.TestFullLogReconstruction.tearDownAfterClass(TestFullLogReconstruction.java:68) > In here... > https://builds.apache.org/job/PreCommit-HBASE-Build/11363/testReport/org.apache.hadoop.hbase/TestFullLogReconstruction/org_apache_hadoop_hbase_TestFullLogReconstruction/ > ... see here. > Thread 1250 (RS_CLOSE_META-edd281aedb18:59863-0): > State: TIMED_WAITING > Blocked count: 92 > Waited count: 278 > Stack: > java.lang.Object.wait(Native Method) > > org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) > > org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) > > org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) > > org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) > > org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeFlushMarker(WALUtil.java:81) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) > org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) > org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) > > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748) > We missed a signal? We need to do an interrupt? The log is not all there in > hadoopqa builds so hard to see all that is going on. This test is not in the > flakey set either -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19926) Use a separated class to implement the WALActionListener for Replication
[
https://issues.apache.org/jira/browse/HBASE-19926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351637#comment-16351637
]
Hadoop QA commented on HBASE-19926:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
8s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
44s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
39s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
18m 17s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 99m
7s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}135m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19926 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12909125/HBASE-19926-v1.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 125c8ae8c961 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / bfd74686c7 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/11379/testReport/ |
| Max. process+thread count | 5331 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/11379/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Use a separated class to implement the WALActionListener for Replication
>
>
>
[jira] [Commented] (HBASE-19927) TestFullLogReconstruction flakey
[ https://issues.apache.org/jira/browse/HBASE-19927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351636#comment-16351636 ] stack commented on HBASE-19927: --- Thanks [~Apache9] Will take a look tomorrow probably at this stage. Always interested in problems in shutdown (smile). > TestFullLogReconstruction flakey > > > Key: HBASE-19927 > URL: https://issues.apache.org/jira/browse/HBASE-19927 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: stack >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19927.patch, js, out > > > Fails pretty frequently in hadoopqa builds. > There is a recent hang in > org.apache.hadoop.hbase.TestFullLogReconstruction.tearDownAfterClass(TestFullLogReconstruction.java:68) > In here... > https://builds.apache.org/job/PreCommit-HBASE-Build/11363/testReport/org.apache.hadoop.hbase/TestFullLogReconstruction/org_apache_hadoop_hbase_TestFullLogReconstruction/ > ... see here. > Thread 1250 (RS_CLOSE_META-edd281aedb18:59863-0): > State: TIMED_WAITING > Blocked count: 92 > Waited count: 278 > Stack: > java.lang.Object.wait(Native Method) > > org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) > > org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) > > org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) > > org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) > > org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeFlushMarker(WALUtil.java:81) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) > org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) > org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) > > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748) > We missed a signal? We need to do an interrupt? The log is not all there in > hadoopqa builds so hard to see all that is going on. This test is not in the > flakey set either -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19914) Refactor TestVisibilityLabelsOnNewVersionBehaviorTable
[ https://issues.apache.org/jira/browse/HBASE-19914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351638#comment-16351638 ] stack commented on HBASE-19914: --- Go for it. Not following what is going on here but on skim patch is grand. Thanks. > Refactor TestVisibilityLabelsOnNewVersionBehaviorTable > -- > > Key: HBASE-19914 > URL: https://issues.apache.org/jira/browse/HBASE-19914 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19914-v1.patch, HBASE-19914-v2.patch, > HBASE-19914.patch, HBASE-19914.patch > > > And both TestVisibilityLabelsOnNewVersionBehaviorTable and its parent class > run about 2 minutes, which is not safe to declared as MediumTests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19927) TestFullLogReconstruction flakey
[ https://issues.apache.org/jira/browse/HBASE-19927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351640#comment-16351640 ] Duo Zhang commented on HBASE-19927: --- Wait for the region server to crash before reading, tried several times, all passed. And I think the problem is that, we enter the normal shutdown first, scheduled region closing, and then wait on all the regions to be closed. And during the closing, master moves the WAL directory so we will fail when writing flush marker, and AsyncFSWAL will trigger a log roll. And the LogRoller will get an exception when creating new log file and then call rs.abort. I've already done a hack in AsyncFSWAL, that when we get 'Parent directory doesn't exist:' we will fail all the pending requests. But here it is not enough. We can only fail some of the flush maker writing, if there are still flush requests that write flush marker after the failure of log roll, it will be stuck for ever because we will not schedule a consumer to write it out, so we will not trigger another log roll and can not trigger the hack again... A possible solution maybe that, when s log roller finds that it could not roll a WAL, before aborting the RS, shutdown the WAL directly first... > TestFullLogReconstruction flakey > > > Key: HBASE-19927 > URL: https://issues.apache.org/jira/browse/HBASE-19927 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: stack >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19927.patch, js, out > > > Fails pretty frequently in hadoopqa builds. > There is a recent hang in > org.apache.hadoop.hbase.TestFullLogReconstruction.tearDownAfterClass(TestFullLogReconstruction.java:68) > In here... > https://builds.apache.org/job/PreCommit-HBASE-Build/11363/testReport/org.apache.hadoop.hbase/TestFullLogReconstruction/org_apache_hadoop_hbase_TestFullLogReconstruction/ > ... see here. > Thread 1250 (RS_CLOSE_META-edd281aedb18:59863-0): > State: TIMED_WAITING > Blocked count: 92 > Waited count: 278 > Stack: > java.lang.Object.wait(Native Method) > > org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) > > org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) > > org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) > > org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) > > org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeFlushMarker(WALUtil.java:81) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) > org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) > org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) > > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748) > We missed a signal? We need to do an interrupt? The log is not all there in > hadoopqa builds so hard to see all that is going on. This test is not in the > flakey set either -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19927) TestFullLogReconstruction flakey
[ https://issues.apache.org/jira/browse/HBASE-19927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351643#comment-16351643 ] stack commented on HBASE-19927: --- Sounds fun. Sounds like issues we used have with the disruptor having to stuff 'signaling' edits into the pipe so stuff could get woken up to notice the shutdown state. Would be sweet if you took care of the async wal stuff. I'll take a look at the sequencing going down to see if stuff we can improve on. Thanks [~Apache9] > TestFullLogReconstruction flakey > > > Key: HBASE-19927 > URL: https://issues.apache.org/jira/browse/HBASE-19927 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: stack >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19927.patch, js, out > > > Fails pretty frequently in hadoopqa builds. > There is a recent hang in > org.apache.hadoop.hbase.TestFullLogReconstruction.tearDownAfterClass(TestFullLogReconstruction.java:68) > In here... > https://builds.apache.org/job/PreCommit-HBASE-Build/11363/testReport/org.apache.hadoop.hbase/TestFullLogReconstruction/org_apache_hadoop_hbase_TestFullLogReconstruction/ > ... see here. > Thread 1250 (RS_CLOSE_META-edd281aedb18:59863-0): > State: TIMED_WAITING > Blocked count: 92 > Waited count: 278 > Stack: > java.lang.Object.wait(Native Method) > > org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) > > org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) > > org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) > > org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) > > org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeFlushMarker(WALUtil.java:81) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) > org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) > org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) > > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748) > We missed a signal? We need to do an interrupt? The log is not all there in > hadoopqa builds so hard to see all that is going on. This test is not in the > flakey set either -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19927) TestFullLogReconstruction flakey
[ https://issues.apache.org/jira/browse/HBASE-19927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351644#comment-16351644 ] Duo Zhang commented on HBASE-19927: --- Let me open another issue to track this. Thanks [~stack]. > TestFullLogReconstruction flakey > > > Key: HBASE-19927 > URL: https://issues.apache.org/jira/browse/HBASE-19927 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: stack >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19927.patch, js, out > > > Fails pretty frequently in hadoopqa builds. > There is a recent hang in > org.apache.hadoop.hbase.TestFullLogReconstruction.tearDownAfterClass(TestFullLogReconstruction.java:68) > In here... > https://builds.apache.org/job/PreCommit-HBASE-Build/11363/testReport/org.apache.hadoop.hbase/TestFullLogReconstruction/org_apache_hadoop_hbase_TestFullLogReconstruction/ > ... see here. > Thread 1250 (RS_CLOSE_META-edd281aedb18:59863-0): > State: TIMED_WAITING > Blocked count: 92 > Waited count: 278 > Stack: > java.lang.Object.wait(Native Method) > > org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:133) > > org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:718) > > org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:605) > > org.apache.hadoop.hbase.regionserver.wal.WALUtil.doFullAppendTransaction(WALUtil.java:154) > > org.apache.hadoop.hbase.regionserver.wal.WALUtil.writeFlushMarker(WALUtil.java:81) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2356) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2328) > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2319) > org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1531) > org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1437) > > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104) > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > java.lang.Thread.run(Thread.java:748) > We missed a signal? We need to do an interrupt? The log is not all there in > hadoopqa builds so hard to see all that is going on. This test is not in the > flakey set either -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19929) Call RS.stop on a session expired RS may hang
[ https://issues.apache.org/jira/browse/HBASE-19929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351645#comment-16351645 ] Duo Zhang commented on HBASE-19929: --- [~stack] FYI. > Call RS.stop on a session expired RS may hang > - > > Key: HBASE-19929 > URL: https://issues.apache.org/jira/browse/HBASE-19929 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Priority: Major > > See the discussion in HBASE-19927. The problem is that, for a normal stop we > will try to close all the regions and wait until they are all closed. But if > the RS has already session expired, master will start the failover work which > will move the WAL directory, and then we will be stuck in writing flush > marker. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-19929) Call RS.stop on a session expired RS may hang
Duo Zhang created HBASE-19929: - Summary: Call RS.stop on a session expired RS may hang Key: HBASE-19929 URL: https://issues.apache.org/jira/browse/HBASE-19929 Project: HBase Issue Type: Bug Reporter: Duo Zhang See the discussion in HBASE-19927. The problem is that, for a normal stop we will try to close all the regions and wait until they are all closed. But if the RS has already session expired, master will start the failover work which will move the WAL directory, and then we will be stuck in writing flush marker. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (HBASE-19726) Failed to start HMaster due to infinite retrying on meta assign
[
https://issues.apache.org/jira/browse/HBASE-19726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack reopened HBASE-19726:
---
Reopen to push an addendum.
TestMetaWithReplicas#testShutdownHandling was failing 100% of the time after
this patch went in. We'd missed a place where we asked about hbase:meta table
state. It was doing the RPC rather than returning the canned response ENABLED.
> Failed to start HMaster due to infinite retrying on meta assign
> ---
>
> Key: HBASE-19726
> URL: https://issues.apache.org/jira/browse/HBASE-19726
> Project: HBase
> Issue Type: Bug
>Reporter: Duo Zhang
>Assignee: stack
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19726.patch
>
>
> This is what I got at first, an exception when trying to write something to
> meta when meta has not been onlined yet.
> {noformat}
> 2018-01-07,21:03:14,389 INFO org.apache.hadoop.hbase.master.HMaster: Running
> RecoverMetaProcedure to ensure proper hbase:meta deploy.
> 2018-01-07,21:03:14,637 INFO
> org.apache.hadoop.hbase.master.procedure.RecoverMetaProcedure: Start pid=1,
> state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure
> failedMetaServer=null, splitWal=true
> 2018-01-07,21:03:14,645 INFO org.apache.hadoop.hbase.master.MasterWalManager:
> Log folder
> hdfs://c402tst-community/hbase/c402tst-community/WALs/c4-hadoop-tst-st27.bj,38900,1515330173896
> belongs to an existing region server
> 2018-01-07,21:03:14,646 INFO org.apache.hadoop.hbase.master.MasterWalManager:
> Log folder
> hdfs://c402tst-community/hbase/c402tst-community/WALs/c4-hadoop-tst-st29.bj,38900,1515330177232
> belongs to an existing region server
> 2018-01-07,21:03:14,648 INFO
> org.apache.hadoop.hbase.master.procedure.RecoverMetaProcedure: pid=1,
> state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure
> failedMetaServer=null, splitWal=true; Retaining meta assignment to server=null
> 2018-01-07,21:03:14,653 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Initialized
> subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE;
> AssignProcedure table=hbase:meta, region=1588230740}]
> 2018-01-07,21:03:14,660 INFO
> org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler: pid=2,
> ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
> table=hbase:meta, region=1588230740 hbase:meta hbase:meta,,1.1588230740
> 2018-01-07,21:03:14,663 INFO
> org.apache.hadoop.hbase.master.assignment.AssignProcedure: Start pid=2,
> ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
> table=hbase:meta, region=1588230740; rit=OFFLINE, location=null;
> forceNewPlan=false, retain=false
> 2018-01-07,21:03:14,831 INFO
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator: Setting hbase:meta
> (replicaId=0) location in ZooKeeper as
> c4-hadoop-tst-st27.bj,38900,1515330173896
> 2018-01-07,21:03:14,841 INFO
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure: Dispatch
> pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure
> table=hbase:meta, region=1588230740; rit=OPENING,
> location=c4-hadoop-tst-st27.bj,38900,1515330173896
> 2018-01-07,21:03:14,992 INFO
> org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher: Using
> procedure batch rpc execution for
> serverName=c4-hadoop-tst-st27.bj,38900,1515330173896 version=3145728
> 2018-01-07,21:03:15,593 ERROR
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: Cannot get replica 0
> location for
> {"totalColumns":1,"row":"hbase:meta","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515330195514}]},"ts":1515330195514}
> 2018-01-07,21:03:15,594 WARN
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure:
> Retryable error trying to transition: pid=2, ppid=1,
> state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=hbase:meta,
> region=1588230740; rit=OPEN,
> location=c4-hadoop-tst-st27.bj,38900,1515330173896
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: IOException: 1 time, servers with issues: null
> at
> org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:54)
> at
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.getErrors(AsyncRequestFutureImpl.java:1250)
> at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:457)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:570)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.put(MetaTableAccessor.java:1450)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.putToMetaTable(MetaTableAccessor.java:1439)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.updateTableState(MetaTableAccessor.java:1785)
> at
> org.apache
[jira] [Resolved] (HBASE-19726) Failed to start HMaster due to infinite retrying on meta assign
[
https://issues.apache.org/jira/browse/HBASE-19726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack resolved HBASE-19726.
---
Resolution: Fixed
Resolving again. Pushed the below addendum to master and branch-2. This should
fix the 100% failing test.
commit b0e998f2a50a50a8d84daa35baff1d4ac99d1c6a
Author: Michael Stack
Date: Sat Feb 3 21:49:42 2018 -0800
HBASE-19726 Failed to start HMaster due to infinite retrying on meta
assign; ADDENDUM Fix failing TestMetaWithReplicas#testShutdownHandling; it was
reading meta TableState
diff --git
a/hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
b/hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
index f80bbc0466..5dc0565162 100644
--- a/hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
@@ -1109,6 +1109,9 @@ public class MetaTableAccessor {
@Nullable
public static TableState getTableState(Connection conn, TableName tableName)
throws IOException {
+if (tableName.equals(TableName.META_TABLE_NAME)) {
+ return new TableState(tableName, TableState.State.ENABLED);
+}
Table metaHTable = getMetaHTable(conn);
Get get = new Get(tableName.getName()).addColumn(getTableFamily(),
getTableStateColumn());
long time = EnvironmentEdgeManager.currentTime();
> Failed to start HMaster due to infinite retrying on meta assign
> ---
>
> Key: HBASE-19726
> URL: https://issues.apache.org/jira/browse/HBASE-19726
> Project: HBase
> Issue Type: Bug
>Reporter: Duo Zhang
>Assignee: stack
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19726.patch
>
>
> This is what I got at first, an exception when trying to write something to
> meta when meta has not been onlined yet.
> {noformat}
> 2018-01-07,21:03:14,389 INFO org.apache.hadoop.hbase.master.HMaster: Running
> RecoverMetaProcedure to ensure proper hbase:meta deploy.
> 2018-01-07,21:03:14,637 INFO
> org.apache.hadoop.hbase.master.procedure.RecoverMetaProcedure: Start pid=1,
> state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure
> failedMetaServer=null, splitWal=true
> 2018-01-07,21:03:14,645 INFO org.apache.hadoop.hbase.master.MasterWalManager:
> Log folder
> hdfs://c402tst-community/hbase/c402tst-community/WALs/c4-hadoop-tst-st27.bj,38900,1515330173896
> belongs to an existing region server
> 2018-01-07,21:03:14,646 INFO org.apache.hadoop.hbase.master.MasterWalManager:
> Log folder
> hdfs://c402tst-community/hbase/c402tst-community/WALs/c4-hadoop-tst-st29.bj,38900,1515330177232
> belongs to an existing region server
> 2018-01-07,21:03:14,648 INFO
> org.apache.hadoop.hbase.master.procedure.RecoverMetaProcedure: pid=1,
> state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure
> failedMetaServer=null, splitWal=true; Retaining meta assignment to server=null
> 2018-01-07,21:03:14,653 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Initialized
> subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE;
> AssignProcedure table=hbase:meta, region=1588230740}]
> 2018-01-07,21:03:14,660 INFO
> org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler: pid=2,
> ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
> table=hbase:meta, region=1588230740 hbase:meta hbase:meta,,1.1588230740
> 2018-01-07,21:03:14,663 INFO
> org.apache.hadoop.hbase.master.assignment.AssignProcedure: Start pid=2,
> ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
> table=hbase:meta, region=1588230740; rit=OFFLINE, location=null;
> forceNewPlan=false, retain=false
> 2018-01-07,21:03:14,831 INFO
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator: Setting hbase:meta
> (replicaId=0) location in ZooKeeper as
> c4-hadoop-tst-st27.bj,38900,1515330173896
> 2018-01-07,21:03:14,841 INFO
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure: Dispatch
> pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure
> table=hbase:meta, region=1588230740; rit=OPENING,
> location=c4-hadoop-tst-st27.bj,38900,1515330173896
> 2018-01-07,21:03:14,992 INFO
> org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher: Using
> procedure batch rpc execution for
> serverName=c4-hadoop-tst-st27.bj,38900,1515330173896 version=3145728
> 2018-01-07,21:03:15,593 ERROR
> org.apache.hadoop.hbase.client.AsyncRequestFutureImpl: Cannot get replica 0
> location for
> {"totalColumns":1,"row":"hbase:meta","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515330195514}]},"ts":1515330195514}
> 2018-01-07,21:03:15,594 WARN
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure:
> Retryable err
[jira] [Commented] (HBASE-19929) Call RS.stop on a session expired RS may hang
[ https://issues.apache.org/jira/browse/HBASE-19929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351649#comment-16351649 ] stack commented on HBASE-19929: --- Good. Yeah. The Master moving the dir to recover a crashed RS is 'normal'; its our primitive fencing mechanism. Can make asyncdfsclient deal with this? Thanks [~Apache9] > Call RS.stop on a session expired RS may hang > - > > Key: HBASE-19929 > URL: https://issues.apache.org/jira/browse/HBASE-19929 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Priority: Major > > See the discussion in HBASE-19927. The problem is that, for a normal stop we > will try to close all the regions and wait until they are all closed. But if > the RS has already session expired, master will start the failover work which > will move the WAL directory, and then we will be stuck in writing flush > marker. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19927) TestFullLogReconstruction flakey
[
https://issues.apache.org/jira/browse/HBASE-19927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351669#comment-16351669
]
Hadoop QA commented on HBASE-19927:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 9m
22s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} hbase-server: The patch generated 0 new + 265
unchanged - 4 fixed = 265 total (was 269) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
35s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
18m 12s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 99m
16s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}147m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19927 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12909128/HBASE-19927.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux b7c85ae0a15f 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / bfd74686c7 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/11380/testReport/ |
| Max. process+thread count | 5367 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/11380/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> TestFullLogReconstruction flakey
>
>
>
[jira] [Resolved] (HBASE-14918) In-Memory MemStore Flush and Compaction
[ https://issues.apache.org/jira/browse/HBASE-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel resolved HBASE-14918. --- Resolution: Done > In-Memory MemStore Flush and Compaction > --- > > Key: HBASE-14918 > URL: https://issues.apache.org/jira/browse/HBASE-14918 > Project: HBase > Issue Type: Umbrella >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: CellBlocksSegmentDesign.pdf, > HBASE-16417-benchmarkresults.pdf, MSLABMove.patch > > > A memstore serves as the in-memory component of a store unit, absorbing all > updates to the store. From time to time these updates are flushed to a file > on disk, where they are compacted (by eliminating redundancies) and > compressed (i.e., written in a compressed format to reduce their storage > size). > We aim to speed up data access, and therefore suggest to apply in-memory > memstore flush. That is to flush the active in-memory segment into an > intermediate buffer where it can be accessed by the application. Data in the > buffer is subject to compaction and can be stored in any format that allows > it to take up smaller space in RAM. The less space the buffer consumes the > longer it can reside in memory before data is flushed to disk, resulting in > better performance. > Specifically, the optimization is beneficial for workloads with > medium-to-high key churn which incur many redundant cells, like persistent > messaging. > We suggest to structure the solution as 4 subtasks (respectively, patches). > (1) Infrastructure - refactoring of the MemStore hierarchy, introducing > segment (StoreSegment) as first-class citizen, and decoupling memstore > scanner from the memstore implementation; > (2) Adding StoreServices facility at the region level to allow memstores > update region counters and access region level synchronization mechanism; > (3) Implementation of a new memstore (CompactingMemstore) with non-optimized > immutable segment representation, and > (4) Memory optimization including compressed format representation and off > heap allocations. > This Jira continues the discussion in HBASE-13408. > Design documents, evaluation results and previous patches can be found in > HBASE-13408. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-17339) Scan-Memory-First Optimization for Get Operations
[ https://issues.apache.org/jira/browse/HBASE-17339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-17339: -- > Scan-Memory-First Optimization for Get Operations > - > > Key: HBASE-17339 > URL: https://issues.apache.org/jira/browse/HBASE-17339 > Project: HBase > Issue Type: Improvement >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-17339-V01.patch, HBASE-17339-V02.patch, > HBASE-17339-V03.patch, HBASE-17339-V03.patch, HBASE-17339-V04.patch, > HBASE-17339-V05.patch, HBASE-17339-V06.patch, read-latency-mixed-workload.jpg > > > The current implementation of a get operation (to retrieve values for a > specific key) scans through all relevant stores of the region; for each store > both memory components (memstores segments) and disk components (hfiles) are > scanned in parallel. > We suggest to apply an optimization that speculatively scans memory-only > components first and only if the result is incomplete scans both memory and > disk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
