date:20180919

[jira] [Commented] (HBASE-21023) Add bypassProcedureToCompletion() API to HbckService

2018-09-19 Thread Hudson (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621537#comment-16621537
 ] 

Hudson commented on HBASE-21023:


Results for branch branch-2.1
[build #350 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/350/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(x) {color:red}-1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/350//General_Nightly_Build_Report/]




(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/350//JDK8_Nightly_Build_Report_(Hadoop2)/]


(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/350//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> Add bypassProcedureToCompletion() API to HbckService
> 
>
> Key: HBASE-21023
> URL: https://issues.apache.org/jira/browse/HBASE-21023
> Project: HBase
>  Issue Type: Sub-task
>  Components: hbck2
>Affects Versions: 2.0.1
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-21023.branch-2.1.001.patch, 
> HBASE-21023.branch-2.1.002.patch, hbase-21023.master.001.patch, 
> hbase-21023.master.001.patch, hbase-21023.master.002.patch
>
>
> completeProcedure/s(): some procedures do not support abort at every step. 
> When these procedures get stuck then they can not be aborted or make further 
> progress. Corrective action is to bypass these procedures to completion.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20941) Create and implement HbckService in master

2018-09-19 Thread stack (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621529#comment-16621529
 ] 

stack commented on HBASE-20941:
---

[~uagashe] I used this new setTableState this evening and I noticed that it 
just writes the state into hbase:meta. It does not update the in-memory state 
in the Master. For me, this meant I had to kill my current Master and go get 
another instance just so the in-memory state agreed w/ hbase:meta. Was 
wondering if you had something mind? I was thinking that we should just call 
TableStateManager#setTableState? What you reckon sir. Here's the patch:

{code}
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
index 6467ea2d57..df2c53327c 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
@@ -2333,8 +2333,9 @@ public class MasterRpcServices extends RSRpcServices
 TableName tn = ProtobufUtil.toTableName(request.getTableName());

 try {
-  HBaseProtos.TableState prevState = MetaTableAccessor.getTableState(conn, 
tn).convert();
-  MetaTableAccessor.updateTableState(conn, tn,
+  HBaseProtos.TableState prevState =
+  this.master.getTableStateManager().getTableState(tn).convert();
+  this.master.getTableStateManager().setTableState(tn,
   TableState.convert(tn, request.getTableState()).getState());
   return 
GetTableStateResponse.newBuilder().setTableState(prevState).build();
 } catch (Exception e) {
{code}

> Create and implement HbckService in master
> --
>
> Key: HBASE-20941
> URL: https://issues.apache.org/jira/browse/HBASE-20941
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.2
>
> Attachments: hbase-20941.master.001.patch, 
> hbase-20941.master.002.patch, hbase-20941.master.003.patch, 
> hbase-20941.master.004.patch, hbase-20941.master.004.patch, 
> hbase-20941.master.004.patch
>
>
> Create HbckService in master and implement following methods:
>  # setTableState(): If table state are inconsistent with action/ procedures 
> working on them, sometimes manipulating their states in meta fix things.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21204) NPE when scan raw DELETE_FAMILY_VERSION and codec is not set

2018-09-19 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621510#comment-16621510
 ] 

Hadoop QA commented on HBASE-21204:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
 6s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m 
57s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
41s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
 6s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  6m 
23s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
2s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 20m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
11s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}  
9m 51s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4 
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}  
1m 31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  6m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
38s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
23s{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}120m 
40s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
58s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}211m 44s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:42ca976 |
| JIRA Issue | HBASE-21204 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12940523/HBASE-21204.branch-2.002.patch
 |
| Optional Tests |  asflicense  cc  unit  hbaseprotoc  javac  javadoc  findbugs 
 shadedjars  hadoopcheck  hbaseanti  checkstyle

[jira] [Comment Edited] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner

2018-09-19 Thread Lars Hofhansl (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621508#comment-16621508
 ] 

Lars Hofhansl edited comment on HBASE-13082 at 9/20/18 4:51 AM:


Interestingly we just upgraded one of our clusters to 1.3.2 and now we see:

"Can't archive compacted file  because of either isCompactedAway = true or 
file has reference, isReferencedInReads = true, skipping for now."

I looked through the code and it seems impossible to visually track where all 
we create StoreFileScanners and whether we always close them.

I don't like reference counting... In this case you forget to close just one 
StoreFileScanner, and BOOM compactions now can *never* succeed;
 unless and until you move the region or bounce the region server. Not good!

[~ram_krish], [~anoop.hbase], [~apurtell]


was (Author: lhofhansl):
Interestingly we just upgraded one of our clusters to 1.3.2 and now we see:

"Can't archive compacted file  because of either isCompactedAway = true or 
file has reference, isReferencedInReads = true, skipping for now."

I looked through the code and it seems impossible to visually track where all 
we create StoreFileScanners and whether we always close them.

I don't like reference counting... In this case you forget to close just one 
StoreFileScanner, and BOOM compactions now can *never* succeed;
unless and until you move the region or bounce the region server. Not good!

 

> Coarsen StoreScanner locks to RegionScanner
> ---
>
> Key: HBASE-13082
> URL: https://issues.apache.org/jira/browse/HBASE-13082
> Project: HBase
>  Issue Type: Bug
>  Components: Performance, Scanners
>Reporter: Lars Hofhansl
>Assignee: ramkrishna.s.vasudevan
>Priority: Major
> Fix For: 1.3.0, 2.0.0
>
> Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt, 
> 13082-v4.txt, 13082.txt, 13082.txt, CountDownLatch-0.98.txt, HBASE-13082.pdf, 
> HBASE-13082_1.pdf, HBASE-13082_12.patch, HBASE-13082_13.patch, 
> HBASE-13082_14.patch, HBASE-13082_15.patch, HBASE-13082_16.patch, 
> HBASE-13082_17.patch, HBASE-13082_18.patch, HBASE-13082_19.patch, 
> HBASE-13082_1_WIP.patch, HBASE-13082_2.pdf, HBASE-13082_2_WIP.patch, 
> HBASE-13082_3.patch, HBASE-13082_4.patch, HBASE-13082_9.patch, 
> HBASE-13082_9.patch, HBASE-13082_withoutpatch.jpg, HBASE-13082_withpatch.jpg, 
> LockVsSynchronized.java, gc.png, gc.png, gc.png, hits.png, next.png, next.png
>
>
> Continuing where HBASE-10015 left of.
> We can avoid locking (and memory fencing) inside StoreScanner by deferring to 
> the lock already held by the RegionScanner.
> In tests this shows quite a scan improvement and reduced CPU (the fences make 
> the cores wait for memory fetches).
> There are some drawbacks too:
> * All calls to RegionScanner need to be remain synchronized
> * Implementors of coprocessors need to be diligent in following the locking 
> contract. For example Phoenix does not lock RegionScanner.nextRaw() and 
> required in the documentation (not picking on Phoenix, this one is my fault 
> as I told them it's OK)
> * possible starving of flushes and compaction with heavy read load. 
> RegionScanner operations would keep getting the locks and the 
> flushes/compactions would not be able finalize the set of files.
> I'll have a patch soon.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner

2018-09-19 Thread Lars Hofhansl (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621508#comment-16621508
 ] 

Lars Hofhansl commented on HBASE-13082:
---

Interestingly we just upgraded one of our clusters to 1.3.2 and now we see:

"Can't archive compacted file  because of either isCompactedAway = true or 
file has reference, isReferencedInReads = true, skipping for now."

I looked through the code and it seems impossible to visually track where all 
we create StoreFileScanners and whether we always close them.

I don't like reference counting... In this case you forget to close just one 
StoreFileScanner, and BOOM compactions now can *never* succeed;
unless and until you move the region or bounce the region server. Not good!

 

> Coarsen StoreScanner locks to RegionScanner
> ---
>
> Key: HBASE-13082
> URL: https://issues.apache.org/jira/browse/HBASE-13082
> Project: HBase
>  Issue Type: Bug
>  Components: Performance, Scanners
>Reporter: Lars Hofhansl
>Assignee: ramkrishna.s.vasudevan
>Priority: Major
> Fix For: 1.3.0, 2.0.0
>
> Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt, 
> 13082-v4.txt, 13082.txt, 13082.txt, CountDownLatch-0.98.txt, HBASE-13082.pdf, 
> HBASE-13082_1.pdf, HBASE-13082_12.patch, HBASE-13082_13.patch, 
> HBASE-13082_14.patch, HBASE-13082_15.patch, HBASE-13082_16.patch, 
> HBASE-13082_17.patch, HBASE-13082_18.patch, HBASE-13082_19.patch, 
> HBASE-13082_1_WIP.patch, HBASE-13082_2.pdf, HBASE-13082_2_WIP.patch, 
> HBASE-13082_3.patch, HBASE-13082_4.patch, HBASE-13082_9.patch, 
> HBASE-13082_9.patch, HBASE-13082_withoutpatch.jpg, HBASE-13082_withpatch.jpg, 
> LockVsSynchronized.java, gc.png, gc.png, gc.png, hits.png, next.png, next.png
>
>
> Continuing where HBASE-10015 left of.
> We can avoid locking (and memory fencing) inside StoreScanner by deferring to 
> the lock already held by the RegionScanner.
> In tests this shows quite a scan improvement and reduced CPU (the fences make 
> the cores wait for memory fetches).
> There are some drawbacks too:
> * All calls to RegionScanner need to be remain synchronized
> * Implementors of coprocessors need to be diligent in following the locking 
> contract. For example Phoenix does not lock RegionScanner.nextRaw() and 
> required in the documentation (not picking on Phoenix, this one is my fault 
> as I told them it's OK)
> * possible starving of flushes and compaction with heavy read load. 
> RegionScanner operations would keep getting the locks and the 
> flushes/compactions would not be able finalize the set of files.
> I'll have a patch soon.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21023) Add bypassProcedureToCompletion() API to HbckService

2018-09-19 Thread Hudson (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621487#comment-16621487
 ] 

Hudson commented on HBASE-21023:


Results for branch branch-2
[build #1274 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1274/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1274//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1274//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1274//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> Add bypassProcedureToCompletion() API to HbckService
> 
>
> Key: HBASE-21023
> URL: https://issues.apache.org/jira/browse/HBASE-21023
> Project: HBase
>  Issue Type: Sub-task
>  Components: hbck2
>Affects Versions: 2.0.1
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-21023.branch-2.1.001.patch, 
> HBASE-21023.branch-2.1.002.patch, hbase-21023.master.001.patch, 
> hbase-21023.master.001.patch, hbase-21023.master.002.patch
>
>
> completeProcedure/s(): some procedures do not support abort at every step. 
> When these procedures get stuck then they can not be aborted or make further 
> progress. Corrective action is to bypass these procedures to completion.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21206) Scan with batch size may return incomplete cells

2018-09-19 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621464#comment-16621464
 ] 

Hadoop QA commented on HBASE-21206:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
55s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
39s{color} | {color:green} branch-1 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
38s{color} | {color:green} branch-1 passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
23s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  2m 
41s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
30s{color} | {color:green} branch-1 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
37s{color} | {color:green} branch-1 passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  2m 
38s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}  
1m 36s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
34s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}110m 48s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
23s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}130m  0s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestEndToEndSplitTransaction |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:61288f8 |
| JIRA Issue | HBASE-21206 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12940524/HBASE-21206.branch-1.v1.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 6b697e241e3d 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9

[jira] [Commented] (HBASE-21208) Bytes#toShort doesn't work without unsafe

2018-09-19 Thread Reid Chan (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621426#comment-16621426
 ] 

Reid Chan commented on HBASE-21208:
---

*{color:green}+1{color}* for v1

> Bytes#toShort doesn't work without unsafe
> -
>
> Key: HBASE-21208
> URL: https://issues.apache.org/jira/browse/HBASE-21208
> Project: HBase
>  Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21208.v0.patch, HBASE-21208.v1.patch
>
>
> seems we put the brackets in the wrong place.
> {code}
>   short n = 0;
>   n = (short) ((n ^ bytes[offset]) & 0xFF);
>   n = (short) (n << 8);
>   n = (short) ((n ^ bytes[offset+1]) & 0xFF);   // this one
>   return n;
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir

2018-09-19 Thread Reid Chan (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621424#comment-16621424
 ] 

Reid Chan commented on HBASE-20734:
---

Good to go by my judge, other committers may not have enough cycles.
You can commit your own, or i can do your favor.


> Colocate recovered edits directory with hbase.wal.dir
> -
>
> Key: HBASE-20734
> URL: https://issues.apache.org/jira/browse/HBASE-20734
> Project: HBase
>  Issue Type: Improvement
>  Components: MTTR, Recovery, wal
>Reporter: Ted Yu
>Assignee: Zach York
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20734.branch-1.001.patch, 
> HBASE-20734.branch-1.002.patch, HBASE-20734.branch-1.003.patch, 
> HBASE-20734.branch-1.004.patch, HBASE-20734.branch-1.005.patch, 
> HBASE-20734.master.001.patch, HBASE-20734.master.002.patch, 
> HBASE-20734.master.003.patch, HBASE-20734.master.004.patch, 
> HBASE-20734.master.005.patch, HBASE-20734.master.006.patch, 
> HBASE-20734.master.007.patch, HBASE-20734.master.008.patch, 
> HBASE-20734.master.009.patch, HBASE-20734.master.010.patch, 
> HBASE-20734.master.011.patch, HBASE-20734.master.012.patch
>
>
> During investigation of HBASE-20723, I realized that we wouldn't get the best 
> performance when hbase.wal.dir is configured to be on different (fast) media 
> than hbase rootdir w.r.t. recovered edits since recovered edits directory is 
> currently under rootdir.
> Such setup may not result in fast recovery when there is region server 
> failover.
> This issue is to find proper (hopefully backward compatible) way in 
> colocating recovered edits directory with hbase.wal.dir .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21035) Meta Table should be able to online even if all procedures are lost

2018-09-19 Thread Duo Zhang (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621413#comment-16621413
 ] 

Duo Zhang commented on HBASE-21035:
---

Maybe a flag to indicate whether to persist? And reset it every time.

> Meta Table should be able to online even if all procedures are lost
> ---
>
> Key: HBASE-21035
> URL: https://issues.apache.org/jira/browse/HBASE-21035
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.1.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Attachments: HBASE-21035.branch-2.0.001.patch, 
> HBASE-21035.branch-2.1.001.patch
>
>
> After HBASE-20708, we changed the way we init after master starts. It will 
> only check WAL dirs and compare to Zookeeper RS nodes to decide which server 
> need to expire. For servers which's dir is ending with 'SPLITTING', we assure 
> that there will be a SCP for it.
> But, if the server with the meta region crashed before master restarts, and 
> if all the procedure wals are lost (due to bug, or deleted manually, 
> whatever), the new restarted master will be stuck when initing. Since no one 
> will bring meta region online.
> Although it is an anomaly case, but I think no matter what happens, we need 
> to online meta region. Otherwise, we are sitting ducks, noting can be done.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21206) Scan with batch size may return incomplete cells

2018-09-19 Thread Zheng Hu (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-21206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zheng Hu updated HBASE-21206:
-
Attachment: HBASE-21206.branch-1.v1.patch

> Scan with batch size may return incomplete cells
> 
>
> Key: HBASE-21206
> URL: https://issues.apache.org/jira/browse/HBASE-21206
> Project: HBase
>  Issue Type: Bug
>  Components: scan
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8, 2.1.1, 2.0.3
>
> Attachments: HBASE-21206.branch-1.v1.patch, HBASE-21206.v1.patch, 
> HBASE-21206.v1.patch, HBASE-21206.v2.patch, HBASE-21206.v3.patch, ut.patch
>
>
> See the attached UT.  the table has 5 columns and each column has at least 
> one cell in it, but when we scan the table with batchSize=3,  we only got 3 
> cells returned , the other 2 cells got lost ...
> It's a critial bug and should be fixed..



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21204) NPE when scan raw DELETE_FAMILY_VERSION and codec is not set

2018-09-19 Thread Jingyun Tian (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jingyun Tian updated HBASE-21204:
-
Attachment: HBASE-21204.branch-2.002.patch

> NPE when scan raw DELETE_FAMILY_VERSION and codec is not set
> 
>
> Key: HBASE-21204
> URL: https://issues.apache.org/jira/browse/HBASE-21204
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.1.0, 2.0.0, 2.2.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Fix For: 2.1.0, 2.0.0, 2.2.0
>
> Attachments: HBASE-21204.branch-2.001.patch, 
> HBASE-21204.branch-2.002.patch, HBASE-21204.master.001.patch, 
> HBASE-21204.master.002.patch, HBASE-21204.master.003.patch
>
>
> There are 7 types of our Cell,
> Minimum((byte)0),
> Put((byte)4),
> Delete((byte)8),
> DeleteFamilyVersion((byte)10),
> DeleteColumn((byte)12),
> DeleteFamily((byte)14),
> Maximum((byte)255);
> But there are only 6 types of our CellType protobuf definition:
> enum CellType {
> MINIMUM = 0;
> PUT = 4;
> DELETE = 8;
> DELETE_FAMILY_VERSION = 10;
> DELETE_COLUMN = 12;
> DELETE_FAMILY = 14;
> MAXIMUM = 255;
> }
> Thus if we scan raw data which is DELETE_FAMILY_VERSION，it will throw NPE.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HBASE-21210) Add bypassProcedure() API to HBCK2

2018-09-19 Thread stack (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-21210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-21210.
---
  Resolution: Fixed
Release Note: 
Added a bypass to hbck2:

{code}
$ 
HBASE_CLASSPATH_PREFIX=../hbase-operator-tools/hbase-hbck2/target/hbase-hbck2-1.0.0-SNAPSHOT.jar
 ./bin/hbase org.apache.hbase.HBCK2
usage: HBCK2 [OPTIONS] COMMAND 

Options:
 -d,--debug run with debug output
 -h,--help  output this help message
 -p,--hbase.zookeeper.property.clientPort   peerport of target hbase
ensemble
 -q,--hbase.zookeeper.quorum   ensemble of target hbase
 -z,--zookeeper.znode.parentparent znode of target hbase

Commands:
 setTableState  
   Possible table states: ENABLED, DISABLED, DISABLING, ENABLING
   To read current table state, in the hbase shell run:
 hbase> get 'hbase:meta', '', 'table:state'
   A value of \x08\x00 == ENABLED, \x08\x01 == DISABLED, etc.
   An example making table name 'user' ENABLED:
 $ HBCK2 setTableState users ENABLED
   Returns whatever the previous table state was.

 assigns ...
   A 'raw' assign that can be used even during Master initialization.
   Skirts Coprocessors. Pass one or more encoded RegionNames:
   e.g. 1588230740 is hard-coded encoding for hbase:meta region and
   de00010733901a05f5a2a3a382e27dd4 is an example of what a random
   user-space encoded Region name looks like. For example:
 $ HBCK2 assign 1588230740 de00010733901a05f5a2a3a382e27dd4
   Returns the pid of the created AssignProcedure or -1 if none.

 bypass [OPTIONS] ...
   Pass one (or more) procedure 'pid's to skip to the procedure finish.
   Parent of this procedures will also skip to its finish. Entities will
   be left in an inconsistent state and will require manual fixup.
   Pass --force to break any outstanding locks.
   Pass --waitTime= to wait on entity lock before giving up.
   Default: force=false and waitTime=0. Returns true if succeeded.

 unassigns ...
   A 'raw' unassign that can be used even during Master initialization.
   Skirts Coprocessors. Pass one or more encoded RegionNames:
   Skirts Coprocessors. Pass one or more encoded RegionNames:
   de00010733901a05f5a2a3a382e27dd4 is an example of what a random
   user-space encoded Region name looks like. For example:
 $ HBCK2 unassign 1588230740 de00010733901a05f5a2a3a382e27dd4
   Returns the pid of the created UnassignProcedure or -1 if none.
{code}

Pushed to branch-2.1+

Tested on cluster bypassing thousands of procedures at a time using this 
addtion.

> Add bypassProcedure() API to HBCK2
> --
>
> Key: HBASE-21210
> URL: https://issues.apache.org/jira/browse/HBASE-21210
> Project: HBase
>  Issue Type: Sub-task
>  Components: hbase-operator-tools, hbck2
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.1.1
>
>
> This JIRA is for adding bypassProcedure to HBCK2 over in 
> hbase-operator-tools. Depends on HBASE-21023 being completed first.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21102) ServerCrashProcedure should select target server where no other replicas exist for the current region

2018-09-19 Thread Hudson (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621310#comment-16621310
 ] 

Hudson commented on HBASE-21102:


Results for branch branch-2
[build #1273 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1273/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1273//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1273//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1273//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> ServerCrashProcedure should select target server where no other replicas 
> exist for the current region
> -
>
> Key: HBASE-21102
> URL: https://issues.apache.org/jira/browse/HBASE-21102
> Project: HBase
>  Issue Type: Bug
>  Components: Region Assignment
>Affects Versions: 3.0.0, 2.2.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Major
> Attachments: 21102.addendum2.txt, HBASE-21102_1.patch, 
> HBASE-21102_2.patch, HBASE-21102_3.patch, HBASE-21102_4.patch, 
> HBASE-21102_addendum.patch, HBASE-21102_addendum.patch, 
> HBASE-21102_addendum.patch, HBASE-21102_branch-2.1.patch, 
> HBASE-21102_branch-2.1.patch, HBASE-21102_initial.patch
>
>
> Currently when a server with region replica crashes, when the target server 
> is created for the replica region assignment there is no guarentee that a 
> server is selected where there is no other replica for the current region 
> getting assigned. It so happens that currently we do an assignment randomly 
> and later the LB comes and identifies these cases and again does MOVE for 
> such regions. It will be better if we can identify target servers at least 
> minimally ensuring that replicas are not colocated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21156) [hbck2] Queue an assign of hbase:meta and bulk assign/unassign

2018-09-19 Thread Hudson (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621311#comment-16621311
 ] 

Hudson commented on HBASE-21156:


Results for branch branch-2
[build #1273 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1273/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1273//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1273//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1273//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> [hbck2] Queue an assign of hbase:meta and bulk assign/unassign
> --
>
> Key: HBASE-21156
> URL: https://issues.apache.org/jira/browse/HBASE-21156
> Project: HBase
>  Issue Type: Sub-task
>  Components: hbck2
>Affects Versions: 2.1.0
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.1.1
>
> Attachments: HBASE-21156.branch-2.1.001.patch, 
> HBASE-21156.branch-2.1.002.patch, HBASE-21156.branch-2.1.003.patch, 
> HBASE-21156.branch-2.1.004.patch, HBASE-21156.branch-2.1.005.patch
>
>
> We need this to effect repair when damage.
> If procedure WALs AND a server WAL dir are lost or cleaned or we crashed 
> during partial split (unlikely scenarios but nonetheless possible), a Master 
> can be stuck unable to become active because there is no assign procedure for 
> hbase:meta in the system.
> The reasonable argument over in HBASE-21035 has it that attempts at 
> auto-repair under these extremes could cause other issues so at least until 
> we learn more, we for now punt to the operator for fix-up.
> To reproduce the catastrophe, see notes in HBASE-21035 (and [~allan163]'s 
> test).
> UPDATE: HBASE-21191 adds a Master assuming an "holding-pattern" if on startup 
> it does not have an assign for meta (possible if we lose all Master WAL 
> Procs.). Holding pattern is needed because we were exiting after one minute 
> of RPC'ing to old meta location. To inject an assign, the Admin#assign won't 
> work because it gets rejected because the "Master is Initializing". So we 
> need to be able to assign hbase:meta even if "Master is initializing". Also, 
> while in here, add being able to bulk assign because assigning a 
> Region-at-a-time from the shell only works if the offflined region count is 
> in the low 10s; fails when thousands offline.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21156) [hbck2] Queue an assign of hbase:meta and bulk assign/unassign

2018-09-19 Thread Hudson (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621303#comment-16621303
 ] 

Hudson commented on HBASE-21156:


Results for branch branch-2.1
[build #349 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/349/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/349//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/349//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/349//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(/) {color:green}+1 client integration test{color}


> [hbck2] Queue an assign of hbase:meta and bulk assign/unassign
> --
>
> Key: HBASE-21156
> URL: https://issues.apache.org/jira/browse/HBASE-21156
> Project: HBase
>  Issue Type: Sub-task
>  Components: hbck2
>Affects Versions: 2.1.0
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.1.1
>
> Attachments: HBASE-21156.branch-2.1.001.patch, 
> HBASE-21156.branch-2.1.002.patch, HBASE-21156.branch-2.1.003.patch, 
> HBASE-21156.branch-2.1.004.patch, HBASE-21156.branch-2.1.005.patch
>
>
> We need this to effect repair when damage.
> If procedure WALs AND a server WAL dir are lost or cleaned or we crashed 
> during partial split (unlikely scenarios but nonetheless possible), a Master 
> can be stuck unable to become active because there is no assign procedure for 
> hbase:meta in the system.
> The reasonable argument over in HBASE-21035 has it that attempts at 
> auto-repair under these extremes could cause other issues so at least until 
> we learn more, we for now punt to the operator for fix-up.
> To reproduce the catastrophe, see notes in HBASE-21035 (and [~allan163]'s 
> test).
> UPDATE: HBASE-21191 adds a Master assuming an "holding-pattern" if on startup 
> it does not have an assign for meta (possible if we lose all Master WAL 
> Procs.). Holding pattern is needed because we were exiting after one minute 
> of RPC'ing to old meta location. To inject an assign, the Admin#assign won't 
> work because it gets rejected because the "Master is Initializing". So we 
> need to be able to assign hbase:meta even if "Master is initializing". Also, 
> while in here, add being able to bulk assign because assigning a 
> Region-at-a-time from the shell only works if the offflined region count is 
> in the low 10s; fails when thousands offline.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21023) Add bypassProcedureToCompletion() API to HbckService

2018-09-19 Thread stack (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-21023:
--
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Pushed to branch-2.1+

> Add bypassProcedureToCompletion() API to HbckService
> 
>
> Key: HBASE-21023
> URL: https://issues.apache.org/jira/browse/HBASE-21023
> Project: HBase
>  Issue Type: Sub-task
>  Components: hbck2
>Affects Versions: 2.0.1
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-21023.branch-2.1.001.patch, 
> HBASE-21023.branch-2.1.002.patch, hbase-21023.master.001.patch, 
> hbase-21023.master.001.patch, hbase-21023.master.002.patch
>
>
> completeProcedure/s(): some procedures do not support abort at every step. 
> When these procedures get stuck then they can not be aborted or make further 
> progress. Corrective action is to bypass these procedures to completion.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-21211) Can't Read Partitions File - Partitions File deleted

2018-09-19 Thread KSHITIJ GAUTAM (JIRA)

KSHITIJ GAUTAM created HBASE-21211:
--

 Summary: Can't Read Partitions File - Partitions File deleted 
 Key: HBASE-21211
 URL: https://issues.apache.org/jira/browse/HBASE-21211
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.5.0, 1.6.0
 Environment: * HBase Version: 1.2.0-cdh5.11.1 (the line that deletes 
the file still exists)
 * hadoop version
 * Hadoop 2.6.0-cdh5.11.1
 * Subversion http://github.com/cloudera/hadoop -r 
b581c269ca3610c603b6d7d1da0d14dfb6684aa3
 * From source with checksum c6cbc4f20a8a571dd7c9f743984da1
 * This command was run using /usr/lib/hadoop/hadoop-common-2.6.0-cdh5.11.1.jar
Reporter: KSHITIJ GAUTAM
 Fix For: 1.5.0, 1.6.0
 Attachments: 
0001-do-not-delete-the-partitions-file-if-the-session-is-.patch

Hi team, we have a MapReduce job that uses the bulkload option instead of 
direct puts to import data e.g., 
{code:java}
HFileOutputFormat2.configureIncrementalLoad(job, table, locator);{code}
 However we have been running into a situation where partitions file is deleted 
by the termination of the JVM process, where JVM process kicks off the 
MapReduce job but it's also waiting to run the `configureIncrementalLoad` that 
executes the configurePartitioner. 

 

_Error: java.lang.IllegalArgumentException: Can't read partitions file at 
org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:116)_

 

We think the line#827 of 
[HFileOutputFormat2|https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java#L827]
 could be the root cause. 

 
{code:java}
fs.deleteOnExit(partitionsPath);{code}
 

We have created our custom HFileOutputFormat that doesn't delete the partitions 
file and have fixed the problem for our cluster. We propose that a cleanup 
method could be created which deletes the partitions file once all the mappers 
have finished.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20952) Re-visit the WAL API

2018-09-19 Thread Ted Yu (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621260#comment-16621260
 ] 

Ted Yu commented on HBASE-20952:


Thanks for the patience.

I have modified the google doc link above for a high level design.

Thanks to Josh and Sergey for internal reviews.

> Re-visit the WAL API
> 
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an 
> HBase WAL API should look like. What are the primitive calls that we require 
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We 
> should also have a mind for what is happening in the Ratis LogService (but 
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and 
> backup Replication has the use-case for "tail"'ing the WAL which we 
> should provide via our new API. B doesn't do anything fancy (IIRC). We 
> should make sure all consumers are generally going to be OK with the API we 
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods 
> which were "bolted" on such as {{AbstractFSWAL}} and 
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the 
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability 
> annotations are chosen.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21023) Add bypassProcedureToCompletion() API to HbckService

2018-09-19 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621246#comment-16621246
 ] 

Hadoop QA commented on HBASE-21023:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} branch-2.1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
 7s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 
12s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
 6s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  3m 
29s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m 
22s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
20s{color} | {color:green} branch-2.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 16m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m 
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
14s{color} | {color:red} hbase-procedure: The patch generated 2 new + 10 
unchanged - 0 fixed = 12 total (was 10) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  3m 
29s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}  
8m 31s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4 
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}  
1m 38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
30s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m  
7s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m  
2s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}116m 
21s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  1m 
12s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}202m  7s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:42ca976 |
| JIRA Issue | HBASE-21023 |
| JIRA Patch URL |

[jira] [Commented] (HBASE-19121) HBCK for AMv2 (A.K.A HBCK2)

2018-09-19 Thread Sean Busbey (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-19121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621044#comment-16621044
 ] 

Sean Busbey commented on HBASE-19121:
-

sweet. Added some comments around teh cli invocation.

> HBCK for AMv2 (A.K.A HBCK2)
> ---
>
> Key: HBASE-19121
> URL: https://issues.apache.org/jira/browse/HBASE-19121
> Project: HBase
>  Issue Type: Bug
>  Components: hbck
>Reporter: stack
>Assignee: Umesh Agashe
>Priority: Major
> Attachments: hbase-19121.master.001.patch
>
>
> We don't have an hbck for the new AM. Old hbck may actually do damage going 
> against AMv2.
> Fix.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HBASE-19121) HBCK for AMv2 (A.K.A HBCK2)

2018-09-19 Thread stack (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-19121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621032#comment-16621032
 ] 

stack edited comment on HBASE-19121 at 9/19/18 6:52 PM:


Anyone can comment now. Sorry about that.

Would love feedback on tooling [~busbey].

Also, Sean just kicked me that there are not HBASE JIRAs referenced in commits 
to hbase-operator-tools hbck which I think was ok while it was getting its 
initial shape but now there is something there, will fill out issues here for 
changes over in the hbase-operator-tools repo from here on out. HBASE-21023 is 
the first.


was (Author: stack):
Anyone can comment now.

Would love feedback on tooling [~busbey]

> HBCK for AMv2 (A.K.A HBCK2)
> ---
>
> Key: HBASE-19121
> URL: https://issues.apache.org/jira/browse/HBASE-19121
> Project: HBase
>  Issue Type: Bug
>  Components: hbck
>Reporter: stack
>Assignee: Umesh Agashe
>Priority: Major
> Attachments: hbase-19121.master.001.patch
>
>
> We don't have an hbck for the new AM. Old hbck may actually do damage going 
> against AMv2.
> Fix.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-21210) Add bypassProcedure() API to HBCK2

2018-09-19 Thread stack (JIRA)

stack created HBASE-21210:
-

 Summary: Add bypassProcedure() API to HBCK2
 Key: HBASE-21210
 URL: https://issues.apache.org/jira/browse/HBASE-21210
 Project: HBase
  Issue Type: Sub-task
  Components: hbase-operator-tools, hbck2
Reporter: stack
Assignee: stack
 Fix For: 2.1.1


This JIRA is for adding bypassProcedure to HBCK2 over in hbase-operator-tools. 
Depends on HBASE-21023 being completed first.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19121) HBCK for AMv2 (A.K.A HBCK2)

2018-09-19 Thread stack (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-19121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621032#comment-16621032
 ] 

stack commented on HBASE-19121:
---

Anyone can comment now.

Would love feedback on tooling [~busbey]

> HBCK for AMv2 (A.K.A HBCK2)
> ---
>
> Key: HBASE-19121
> URL: https://issues.apache.org/jira/browse/HBASE-19121
> Project: HBase
>  Issue Type: Bug
>  Components: hbck
>Reporter: stack
>Assignee: Umesh Agashe
>Priority: Major
> Attachments: hbase-19121.master.001.patch
>
>
> We don't have an hbck for the new AM. Old hbck may actually do damage going 
> against AMv2.
> Fix.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19121) HBCK for AMv2 (A.K.A HBCK2)

2018-09-19 Thread Sean Busbey (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-19121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621029#comment-16621029
 ] 

Sean Busbey commented on HBASE-19121:
-

The google doc only has view access. can you enable commenting access? Or would 
you prefer feedback here?

I'd like to talk about improving this (from the 
[hbase-operator-tools/hbck2/README|https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2#running-hbck2]):

{quote}
{{org.apache.hbase.HBCK2}} is the name of the main class. Running the below 
will dump out the HBCK2 usage:
{code}
 $ HBASE_CLASSPATH_PREFIX=/tmp/hbase-hbck2-1.0.0-SNAPSHOT.jar ./bin/hbase 
org.apache.hbase.HBCK2
{code}
{quote}

> HBCK for AMv2 (A.K.A HBCK2)
> ---
>
> Key: HBASE-19121
> URL: https://issues.apache.org/jira/browse/HBASE-19121
> Project: HBase
>  Issue Type: Bug
>  Components: hbck
>Reporter: stack
>Assignee: Umesh Agashe
>Priority: Major
> Attachments: hbase-19121.master.001.patch
>
>
> We don't have an hbck for the new AM. Old hbck may actually do damage going 
> against AMv2.
> Fix.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21023) Add bypassProcedureToCompletion() API to HbckService

2018-09-19 Thread stack (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-21023:
--
Attachment: HBASE-21023.branch-2.1.002.patch

> Add bypassProcedureToCompletion() API to HbckService
> 
>
> Key: HBASE-21023
> URL: https://issues.apache.org/jira/browse/HBASE-21023
> Project: HBase
>  Issue Type: Sub-task
>  Components: hbck2
>Affects Versions: 2.0.1
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-21023.branch-2.1.001.patch, 
> HBASE-21023.branch-2.1.002.patch, hbase-21023.master.001.patch, 
> hbase-21023.master.001.patch, hbase-21023.master.002.patch
>
>
> completeProcedure/s(): some procedures do not support abort at every step. 
> When these procedures get stuck then they can not be aborted or make further 
> progress. Corrective action is to bypass these procedures to completion.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21164) reportForDuty to spew less log if master is initializing

2018-09-19 Thread Mingliang Liu (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620940#comment-16620940
 ] 

Mingliang Liu commented on HBASE-21164:
---

Yes sir. 

> reportForDuty to spew less log if master is initializing
> 
>
> Key: HBASE-21164
> URL: https://issues.apache.org/jira/browse/HBASE-21164
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: stack
>Assignee: Mingliang Liu
>Priority: Minor
> Attachments: HBASE-21164.005.patch, HBASE-21164.006.patch, 
> HBASE-21164.007.patch, HBASE-21164.008.patch, HBASE-21164.009.patch, 
> HBASE-21164.branch-2.1.001.patch, HBASE-21164.branch-2.1.002.patch, 
> HBASE-21164.branch-2.1.003.patch, HBASE-21164.branch-2.1.004.patch
>
>
> RegionServers do reportForDuty on startup to tell Master they are available. 
> If Master is initializing, and especially on a big cluster when it can take a 
> while particularly if something is amiss, the log every three seconds is 
> annoying and doesn't do anything of use. We should spew less those logs. Here 
> is example:
> {code:java}
> 2018-09-06 14:01:39,312 INFO 
> org.apache.hadoop.hbase.regionserver.HRegionServer: reportForDuty to 
> master=vc0207.halxg.cloudera.com,22001,1536266763109 with port=22001, 
> startcode=1536266763109
> 2018-09-06 14:01:39,312 WARN 
> org.apache.hadoop.hbase.regionserver.HRegionServer: reportForDuty failed; 
> sleeping and then retrying.
> 
> {code}
> For example, I am looking at a large cluster now that had a backlog of 
> procedure WALs. It is taking a couple of hours recreating the procedure-state 
> because there are millions of procedures outstanding. Meantime, the Master 
> log is just full of the above message – every three seconds...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21156) [hbck2] Queue an assign of hbase:meta and bulk assign/unassign

2018-09-19 Thread stack (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-21156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-21156:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

I pushed this to branch-2.1+

> [hbck2] Queue an assign of hbase:meta and bulk assign/unassign
> --
>
> Key: HBASE-21156
> URL: https://issues.apache.org/jira/browse/HBASE-21156
> Project: HBase
>  Issue Type: Sub-task
>  Components: hbck2
>Affects Versions: 2.1.0
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.1.1
>
> Attachments: HBASE-21156.branch-2.1.001.patch, 
> HBASE-21156.branch-2.1.002.patch, HBASE-21156.branch-2.1.003.patch, 
> HBASE-21156.branch-2.1.004.patch, HBASE-21156.branch-2.1.005.patch
>
>
> We need this to effect repair when damage.
> If procedure WALs AND a server WAL dir are lost or cleaned or we crashed 
> during partial split (unlikely scenarios but nonetheless possible), a Master 
> can be stuck unable to become active because there is no assign procedure for 
> hbase:meta in the system.
> The reasonable argument over in HBASE-21035 has it that attempts at 
> auto-repair under these extremes could cause other issues so at least until 
> we learn more, we for now punt to the operator for fix-up.
> To reproduce the catastrophe, see notes in HBASE-21035 (and [~allan163]'s 
> test).
> UPDATE: HBASE-21191 adds a Master assuming an "holding-pattern" if on startup 
> it does not have an assign for meta (possible if we lose all Master WAL 
> Procs.). Holding pattern is needed because we were exiting after one minute 
> of RPC'ing to old meta location. To inject an assign, the Admin#assign won't 
> work because it gets rejected because the "Master is Initializing". So we 
> need to be able to assign hbase:meta even if "Master is initializing". Also, 
> while in here, add being able to bulk assign because assigning a 
> Region-at-a-time from the shell only works if the offflined region count is 
> in the low 10s; fails when thousands offline.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HBASE-21156) [hbck2] Queue an assign of hbase:meta and bulk assign/unassign

2018-09-19 Thread stack (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620893#comment-16620893
 ] 

stack edited comment on HBASE-21156 at 9/19/18 5:17 PM:


I pushed this to branch-2.1+ after addressing checkstyle issue.


was (Author: stack):
I pushed this to branch-2.1+

> [hbck2] Queue an assign of hbase:meta and bulk assign/unassign
> --
>
> Key: HBASE-21156
> URL: https://issues.apache.org/jira/browse/HBASE-21156
> Project: HBase
>  Issue Type: Sub-task
>  Components: hbck2
>Affects Versions: 2.1.0
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.1.1
>
> Attachments: HBASE-21156.branch-2.1.001.patch, 
> HBASE-21156.branch-2.1.002.patch, HBASE-21156.branch-2.1.003.patch, 
> HBASE-21156.branch-2.1.004.patch, HBASE-21156.branch-2.1.005.patch
>
>
> We need this to effect repair when damage.
> If procedure WALs AND a server WAL dir are lost or cleaned or we crashed 
> during partial split (unlikely scenarios but nonetheless possible), a Master 
> can be stuck unable to become active because there is no assign procedure for 
> hbase:meta in the system.
> The reasonable argument over in HBASE-21035 has it that attempts at 
> auto-repair under these extremes could cause other issues so at least until 
> we learn more, we for now punt to the operator for fix-up.
> To reproduce the catastrophe, see notes in HBASE-21035 (and [~allan163]'s 
> test).
> UPDATE: HBASE-21191 adds a Master assuming an "holding-pattern" if on startup 
> it does not have an assign for meta (possible if we lose all Master WAL 
> Procs.). Holding pattern is needed because we were exiting after one minute 
> of RPC'ing to old meta location. To inject an assign, the Admin#assign won't 
> work because it gets rejected because the "Master is Initializing". So we 
> need to be able to assign hbase:meta even if "Master is initializing". Also, 
> while in here, add being able to bulk assign because assigning a 
> Region-at-a-time from the shell only works if the offflined region count is 
> in the low 10s; fails when thousands offline.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close

2018-09-19 Thread Francis Liu (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Francis Liu updated HBASE-20704:

Fix Version/s: 1.4.8
   1.3.3
   1.5.0

> Sometimes some compacted storefiles are not archived on region close
> 
>
> Key: HBASE-20704
> URL: https://issues.apache.org/jira/browse/HBASE-20704
> Project: HBase
>  Issue Type: Bug
>  Components: Compaction
>Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0
>Reporter: Francis Liu
>Assignee: Francis Liu
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.4.8, 2.1.1, 2.0.3
>
> Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch, 
> HBASE-20704.003.patch, HBASE-20704.004.draft.patch, HBASE-20704.005.patch, 
> HBASE-20704.006.patch, HBASE-20704.007.patch, HBASE-20704.branch-1.001.patch
>
>
> During region close compacted files which have not yet been archived by the 
> discharger are archived as part of the region closing process. It is 
> important that these files are wholly archived to insure data consistency. ie 
> a storefile containing delete tombstones can be archived while older 
> storefiles containing cells that were supposed to be deleted are left 
> unarchived thereby undeleting those cells. 
> On region close a compacted storefile is skipped from archiving if it has 
> read references (ie open scanners). This behavior is correct for when the 
> discharger chore runs but on region close consistency is of course more 
> important so we should add a special case to ignore any references on the 
> storefile and go ahead and archive it. 
> Attached patch contains a unit test that reproduces the problem and the 
> proposed fix.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21206) Scan with batch size may return incomplete cells

2018-09-19 Thread Zheng Hu (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620810#comment-16620810
 ] 

Zheng Hu commented on HBASE-21206:
--

bq. It seems that you replied on the review board and said 'It's reasonable. 
Thanks', but in patch v3 you haven't changed the code? 
No, I've fixed the comment. you can see: 
https://reviews.apache.org/r/68745/diff/3-4/

Upload  HBASE-21206.branch-1.v1.patch for branch-1.4 & branch-1.   Checked the 
code,  branch-1.2 & branch-1.3 didn't has this bug,  because it did not 
consider the batch limit case  as partial result [1] ... at the client side,  
if rpc return two results, the first with 2 cells, the second with 3 cells,  
when scan with batch=3,  the client will just read the first result with 2 
cells, and the next result with 3 cells... it does not collect them into 3 + 
2... the javadoc also indicate this [2].. so we just keep the semantic.

BTW HBASE-21206.v3.patch can be applied to branch-2.0 & branch2.1 & master ..

1. 
https://github.com/apache/hbase/blob/branch-1.2/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java#L2617
2. 
https://github.com/apache/hbase/blob/branch-1.2/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java#L470


> Scan with batch size may return incomplete cells
> 
>
> Key: HBASE-21206
> URL: https://issues.apache.org/jira/browse/HBASE-21206
> Project: HBase
>  Issue Type: Bug
>  Components: scan
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8, 2.1.1, 2.0.3
>
> Attachments: HBASE-21206.v1.patch, HBASE-21206.v1.patch, 
> HBASE-21206.v2.patch, HBASE-21206.v3.patch, ut.patch
>
>
> See the attached UT.  the table has 5 columns and each column has at least 
> one cell in it, but when we scan the table with batchSize=3,  we only got 3 
> cells returned , the other 2 cells got lost ...
> It's a critial bug and should be fixed..



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21206) Scan with batch size may return incomplete cells

2018-09-19 Thread Zheng Hu (JIRA)



 [ 
https://issues.apache.org/jira/browse/HBASE-21206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zheng Hu updated HBASE-21206:
-
Fix Version/s: (was: 1.2.8)
   (was: 1.3.3)

> Scan with batch size may return incomplete cells
> 
>
> Key: HBASE-21206
> URL: https://issues.apache.org/jira/browse/HBASE-21206
> Project: HBase
>  Issue Type: Bug
>  Components: scan
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8, 2.1.1, 2.0.3
>
> Attachments: HBASE-21206.v1.patch, HBASE-21206.v1.patch, 
> HBASE-21206.v2.patch, HBASE-21206.v3.patch, ut.patch
>
>
> See the attached UT.  the table has 5 columns and each column has at least 
> one cell in it, but when we scan the table with batchSize=3,  we only got 3 
> cells returned , the other 2 cells got lost ...
> It's a critial bug and should be fixed..



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21169) Initiate hbck2 tool in hbase-operator-tools repo

2018-09-19 Thread stack (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620791#comment-16620791
 ] 

stack commented on HBASE-21169:
---

[~busbey] makes sense. Let me finish it up

> Initiate hbck2 tool in hbase-operator-tools repo
> 
>
> Key: HBASE-21169
> URL: https://issues.apache.org/jira/browse/HBASE-21169
> Project: HBase
>  Issue Type: Sub-task
>  Components: hbck2
>Affects Versions: 2.1.0
>Reporter: Umesh Agashe
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: hbase-21169.master.001.patch
>
>
> Create hbck2 tool in hbase-operator-tools 
> (https://github.com/apache/hbase-operator-tools.git) repo. This is not 
> intended to be complete tool but initial changes with usage, ability to 
> connect to server, logging, and using newly added HbckService etc. Code 
> changes to address specific use cases can be added later and tool will evolve.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21169) Initiate hbck2 tool in hbase-operator-tools repo

2018-09-19 Thread Sean Busbey (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620782#comment-16620782
 ] 

Sean Busbey commented on HBASE-21169:
-

in hbase-operator-tools could it instead be built into the website/docs for 
that particular repo?

> Initiate hbck2 tool in hbase-operator-tools repo
> 
>
> Key: HBASE-21169
> URL: https://issues.apache.org/jira/browse/HBASE-21169
> Project: HBase
>  Issue Type: Sub-task
>  Components: hbck2
>Affects Versions: 2.1.0
>Reporter: Umesh Agashe
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: hbase-21169.master.001.patch
>
>
> Create hbck2 tool in hbase-operator-tools 
> (https://github.com/apache/hbase-operator-tools.git) repo. This is not 
> intended to be complete tool but initial changes with usage, ability to 
> connect to server, logging, and using newly added HbckService etc. Code 
> changes to address specific use cases can be added later and tool will evolve.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21169) Initiate hbck2 tool in hbase-operator-tools repo

2018-09-19 Thread stack (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620765#comment-16620765
 ] 

stack commented on HBASE-21169:
---

bq. is the design in a state where we could start tracking it in 
dev-support/design-docs

Not yet. I think I'd put it over in hbase-operator-tools though rather than 
here in same relative location.

> Initiate hbck2 tool in hbase-operator-tools repo
> 
>
> Key: HBASE-21169
> URL: https://issues.apache.org/jira/browse/HBASE-21169
> Project: HBase
>  Issue Type: Sub-task
>  Components: hbck2
>Affects Versions: 2.1.0
>Reporter: Umesh Agashe
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: hbase-21169.master.001.patch
>
>
> Create hbck2 tool in hbase-operator-tools 
> (https://github.com/apache/hbase-operator-tools.git) repo. This is not 
> intended to be complete tool but initial changes with usage, ability to 
> connect to server, logging, and using newly added HbckService etc. Code 
> changes to address specific use cases can be added later and tool will evolve.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21002) Create assembly and scripts to start Kafka Proxy

2018-09-19 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620753#comment-16620753
 ] 

ASF GitHub Bot commented on HBASE-21002:


hbasejanitor commented on issue #3: HBASE-21002 make an assembly for 
hbase-connectors
URL: https://github.com/apache/hbase-connectors/pull/3#issuecomment-422848121
 
 
   I think the exception was a classpath issue.  I've trimmed down the 
dependencies.  


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Create assembly and scripts to start Kafka Proxy
> 
>
> Key: HBASE-21002
> URL: https://issues.apache.org/jira/browse/HBASE-21002
> Project: HBase
>  Issue Type: Sub-task
>  Components: hbase-connectors
>Reporter: Mike Wingert
>Assignee: Mike Wingert
>Priority: Minor
>
> Add scripts for running and assembly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] hbasejanitor commented on issue #3: HBASE-21002 make an assembly for hbase-connectors

2018-09-19 Thread GitBox

hbasejanitor commented on issue #3: HBASE-21002 make an assembly for 
hbase-connectors
URL: https://github.com/apache/hbase-connectors/pull/3#issuecomment-422848121
 
 
   I think the exception was a classpath issue.  I've trimmed down the 
dependencies.  


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (HBASE-21206) Scan with batch size may return incomplete cells

2018-09-19 Thread Duo Zhang (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620653#comment-16620653
 ] 

Duo Zhang commented on HBASE-21206:
---

It seems that you replied on the review board and said 'It's reasonable. 
Thanks', but in patch v3 you haven't changed the code?

> Scan with batch size may return incomplete cells
> 
>
> Key: HBASE-21206
> URL: https://issues.apache.org/jira/browse/HBASE-21206
> Project: HBase
>  Issue Type: Bug
>  Components: scan
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 1.4.8, 2.1.1, 2.0.3
>
> Attachments: HBASE-21206.v1.patch, HBASE-21206.v1.patch, 
> HBASE-21206.v2.patch, HBASE-21206.v3.patch, ut.patch
>
>
> See the attached UT.  the table has 5 columns and each column has at least 
> one cell in it, but when we scan the table with batchSize=3,  we only got 3 
> cells returned , the other 2 cells got lost ...
> It's a critial bug and should be fixed..



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21002) Create assembly and scripts to start Kafka Proxy

2018-09-19 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620610#comment-16620610
 ] 

ASF GitHub Bot commented on HBASE-21002:


hbasejanitor commented on a change in pull request #3: HBASE-21002 make an 
assembly for hbase-connectors
URL: https://github.com/apache/hbase-connectors/pull/3#discussion_r218810828
 
 

 ##
 File path: kafka/README
 ##
 @@ -0,0 +1,126 @@
+Hbase Kafka Proxy
 
 Review comment:
   Yes, it should =-)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Create assembly and scripts to start Kafka Proxy
> 
>
> Key: HBASE-21002
> URL: https://issues.apache.org/jira/browse/HBASE-21002
> Project: HBase
>  Issue Type: Sub-task
>  Components: hbase-connectors
>Reporter: Mike Wingert
>Assignee: Mike Wingert
>Priority: Minor
>
> Add scripts for running and assembly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] hbasejanitor commented on a change in pull request #3: HBASE-21002 make an assembly for hbase-connectors

2018-09-19 Thread GitBox

hbasejanitor commented on a change in pull request #3: HBASE-21002 make an 
assembly for hbase-connectors
URL: https://github.com/apache/hbase-connectors/pull/3#discussion_r218810828
 
 

 ##
 File path: kafka/README
 ##
 @@ -0,0 +1,126 @@
+Hbase Kafka Proxy
 
 Review comment:
   Yes, it should =-)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (HBASE-21002) Create assembly and scripts to start Kafka Proxy

2018-09-19 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620603#comment-16620603
 ] 

ASF GitHub Bot commented on HBASE-21002:


hbasejanitor commented on a change in pull request #3: HBASE-21002 make an 
assembly for hbase-connectors
URL: https://github.com/apache/hbase-connectors/pull/3#discussion_r218809228
 
 

 ##
 File path: kafka/hbase-kafka-proxy/pom.xml
 ##
 @@ -76,43 +77,40 @@

   org.apache.avro
   avro
-
+   
 
   org.apache.hbase.connectors
   hbase-kafka-model
 
-
-  org.apache.hbase
-  hbase-common
-
+
 
   org.apache.hbase
   hbase-common
   test-jar
   test
 
-
-  org.apache.hbase
-  hbase-client
-
-
-  org.apache.hbase
-  hbase-zookeeper
-
+
 
   org.apache.hbase
   hbase-server
+  provided
 
 
   org.apache.hbase
   hbase-annotations
+  compile
 
 
   org.apache.hbase
   hbase-annotations
   test-jar
   test
 
+
+  org.apache.yetus
+  audience-annotations
+  ${audience-annotations.version}
+
 
   org.apache.kafka
   kafka-clients
 
 Review comment:
   I'll change this to hbase-shaded-client


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Create assembly and scripts to start Kafka Proxy
> 
>
> Key: HBASE-21002
> URL: https://issues.apache.org/jira/browse/HBASE-21002
> Project: HBase
>  Issue Type: Sub-task
>  Components: hbase-connectors
>Reporter: Mike Wingert
>Assignee: Mike Wingert
>Priority: Minor
>
> Add scripts for running and assembly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] hbasejanitor commented on a change in pull request #3: HBASE-21002 make an assembly for hbase-connectors

2018-09-19 Thread GitBox

hbasejanitor commented on a change in pull request #3: HBASE-21002 make an 
assembly for hbase-connectors
URL: https://github.com/apache/hbase-connectors/pull/3#discussion_r218809228
 
 

 ##
 File path: kafka/hbase-kafka-proxy/pom.xml
 ##
 @@ -76,43 +77,40 @@

   org.apache.avro
   avro
-
+   
 
   org.apache.hbase.connectors
   hbase-kafka-model
 
-
-  org.apache.hbase
-  hbase-common
-
+
 
   org.apache.hbase
   hbase-common
   test-jar
   test
 
-
-  org.apache.hbase
-  hbase-client
-
-
-  org.apache.hbase
-  hbase-zookeeper
-
+
 
   org.apache.hbase
   hbase-server
+  provided
 
 
   org.apache.hbase
   hbase-annotations
+  compile
 
 
   org.apache.hbase
   hbase-annotations
   test-jar
   test
 
+
+  org.apache.yetus
+  audience-annotations
+  ${audience-annotations.version}
+
 
   org.apache.kafka
   kafka-clients
 
 Review comment:
   I'll change this to hbase-shaded-client


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir

2018-09-19 Thread Zach York (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620581#comment-16620581
 ] 

Zach York commented on HBASE-20734:
---

Thanks [~reidchan]!

What is the next step here? Are we still waiting for any other reviews or can 
we get this committed?

> Colocate recovered edits directory with hbase.wal.dir
> -
>
> Key: HBASE-20734
> URL: https://issues.apache.org/jira/browse/HBASE-20734
> Project: HBase
>  Issue Type: Improvement
>  Components: MTTR, Recovery, wal
>Reporter: Ted Yu
>Assignee: Zach York
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20734.branch-1.001.patch, 
> HBASE-20734.branch-1.002.patch, HBASE-20734.branch-1.003.patch, 
> HBASE-20734.branch-1.004.patch, HBASE-20734.branch-1.005.patch, 
> HBASE-20734.master.001.patch, HBASE-20734.master.002.patch, 
> HBASE-20734.master.003.patch, HBASE-20734.master.004.patch, 
> HBASE-20734.master.005.patch, HBASE-20734.master.006.patch, 
> HBASE-20734.master.007.patch, HBASE-20734.master.008.patch, 
> HBASE-20734.master.009.patch, HBASE-20734.master.010.patch, 
> HBASE-20734.master.011.patch, HBASE-20734.master.012.patch
>
>
> During investigation of HBASE-20723, I realized that we wouldn't get the best 
> performance when hbase.wal.dir is configured to be on different (fast) media 
> than hbase rootdir w.r.t. recovered edits since recovered edits directory is 
> currently under rootdir.
> Such setup may not result in fast recovery when there is region server 
> failover.
> This issue is to find proper (hopefully backward compatible) way in 
> colocating recovered edits directory with hbase.wal.dir .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21169) Initiate hbck2 tool in hbase-operator-tools repo

2018-09-19 Thread Sean Busbey (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620572#comment-16620572
 ] 

Sean Busbey commented on HBASE-21169:
-

is the design in a state where we could start tracking it in 
{{dev-support/design-docs}} in the main repo?

> Initiate hbck2 tool in hbase-operator-tools repo
> 
>
> Key: HBASE-21169
> URL: https://issues.apache.org/jira/browse/HBASE-21169
> Project: HBase
>  Issue Type: Sub-task
>  Components: hbck2
>Affects Versions: 2.1.0
>Reporter: Umesh Agashe
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: hbase-21169.master.001.patch
>
>
> Create hbck2 tool in hbase-operator-tools 
> (https://github.com/apache/hbase-operator-tools.git) repo. This is not 
> intended to be complete tool but initial changes with usage, ability to 
> connect to server, logging, and using newly added HbckService etc. Code 
> changes to address specific use cases can be added later and tool will evolve.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21206) Scan with batch size may return incomplete cells

2018-09-19 Thread Mike Drob (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620564#comment-16620564
 ] 

Mike Drob commented on HBASE-21206:
---

Lgtm

> Scan with batch size may return incomplete cells
> 
>
> Key: HBASE-21206
> URL: https://issues.apache.org/jira/browse/HBASE-21206
> Project: HBase
>  Issue Type: Bug
>  Components: scan
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 1.4.8, 2.1.1, 2.0.3
>
> Attachments: HBASE-21206.v1.patch, HBASE-21206.v1.patch, 
> HBASE-21206.v2.patch, HBASE-21206.v3.patch, ut.patch
>
>
> See the attached UT.  the table has 5 columns and each column has at least 
> one cell in it, but when we scan the table with batchSize=3,  we only got 3 
> cells returned , the other 2 cells got lost ...
> It's a critial bug and should be fixed..



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21002) Create assembly and scripts to start Kafka Proxy

2018-09-19 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620531#comment-16620531
 ] 

ASF GitHub Bot commented on HBASE-21002:


hbasejanitor commented on a change in pull request #3: HBASE-21002 make an 
assembly for hbase-connectors
URL: https://github.com/apache/hbase-connectors/pull/3#discussion_r218788287
 
 

 ##
 File path: 
kafka/hbase-kafka-proxy/src/main/java/org/apache/hadoop/hbase/kafka/KafkaProxy.java
 ##
 @@ -154,19 +150,28 @@ public static void main(String[] args) throws Exception {
 options.addOption("e", "enablepeer", false,
 "enable peer on startup (defaults to false)");
 
-
 LOG.info("STARTING executorService " + 
HRegionServer.class.getSimpleName());
 VersionInfo.logVersion();
 
 Configuration conf = HBaseConfiguration.create();
 CommandLine commandLine = null;
+
+Configuration commandLineConf = new Configuration();
 
 Review comment:
   Hi Stack, I'm doing it that way so the user can pass -Dconfkey=confValue on 
the command line.  What I do is create an empty conf object, then use 
GenericOptionsParser to pick out all the -D arguments.  I then feed those 
values into the region server that I start.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Create assembly and scripts to start Kafka Proxy
> 
>
> Key: HBASE-21002
> URL: https://issues.apache.org/jira/browse/HBASE-21002
> Project: HBase
>  Issue Type: Sub-task
>  Components: hbase-connectors
>Reporter: Mike Wingert
>Assignee: Mike Wingert
>Priority: Minor
>
> Add scripts for running and assembly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] hbasejanitor commented on a change in pull request #3: HBASE-21002 make an assembly for hbase-connectors

2018-09-19 Thread GitBox

hbasejanitor commented on a change in pull request #3: HBASE-21002 make an 
assembly for hbase-connectors
URL: https://github.com/apache/hbase-connectors/pull/3#discussion_r218788287
 
 

 ##
 File path: 
kafka/hbase-kafka-proxy/src/main/java/org/apache/hadoop/hbase/kafka/KafkaProxy.java
 ##
 @@ -154,19 +150,28 @@ public static void main(String[] args) throws Exception {
 options.addOption("e", "enablepeer", false,
 "enable peer on startup (defaults to false)");
 
-
 LOG.info("STARTING executorService " + 
HRegionServer.class.getSimpleName());
 VersionInfo.logVersion();
 
 Configuration conf = HBaseConfiguration.create();
 CommandLine commandLine = null;
+
+Configuration commandLineConf = new Configuration();
 
 Review comment:
   Hi Stack, I'm doing it that way so the user can pass -Dconfkey=confValue on 
the command line.  What I do is create an empty conf object, then use 
GenericOptionsParser to pick out all the -D arguments.  I then feed those 
values into the region server that I start.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (HBASE-21002) Create assembly and scripts to start Kafka Proxy

2018-09-19 Thread Mike Wingert (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620527#comment-16620527
 ] 

Mike Wingert commented on HBASE-21002:
--

Hi Stack,

The hbase-client dependency wasn't needed, I'll remove it.

I do have a direct dependency on hbase-server (I start a region server for the 
proxy).  All jars are picked up by the `hbase classpath` command, so there's no 
need to bundle the jar.  My assumption is the hbase config/hbase libraries will 
be available where the proxy is running.

> Create assembly and scripts to start Kafka Proxy
> 
>
> Key: HBASE-21002
> URL: https://issues.apache.org/jira/browse/HBASE-21002
> Project: HBase
>  Issue Type: Sub-task
>  Components: hbase-connectors
>Reporter: Mike Wingert
>Assignee: Mike Wingert
>Priority: Minor
>
> Add scripts for running and assembly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20993) [Auth] IPC client fallback to simple auth allowed doesn't work

2018-09-19 Thread Hudson (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620449#comment-16620449
 ] 

Hudson commented on HBASE-20993:


Results for branch branch-1.4
[build #468 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/468/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(x) {color:red}-1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/468//General_Nightly_Build_Report/]


(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/468//JDK7_Nightly_Build_Report/]


(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/468//JDK8_Nightly_Build_Report_(Hadoop2)/]




(/) {color:green}+1 source release artifact{color}
-- See build output for details.


> [Auth] IPC client fallback to simple auth allowed doesn't work
> --
>
> Key: HBASE-20993
> URL: https://issues.apache.org/jira/browse/HBASE-20993
> Project: HBase
>  Issue Type: Bug
>  Components: Client, IPC/RPC, security
>Affects Versions: 1.2.6, 1.3.2, 1.2.7, 1.4.7
>Reporter: Reid Chan
>Assignee: Jack Bearden
>Priority: Critical
> Fix For: 1.5.0, 1.4.8
>
> Attachments: HBASE-20993.001.patch, 
> HBASE-20993.003.branch-1.flowchart.png, HBASE-20993.branch-1.002.patch, 
> HBASE-20993.branch-1.003.patch, HBASE-20993.branch-1.004.patch, 
> HBASE-20993.branch-1.005.patch, HBASE-20993.branch-1.006.patch, 
> HBASE-20993.branch-1.007.patch, HBASE-20993.branch-1.008.patch, 
> HBASE-20993.branch-1.009.patch, HBASE-20993.branch-1.009.patch, 
> HBASE-20993.branch-1.2.001.patch, HBASE-20993.branch-1.wip.002.patch, 
> HBASE-20993.branch-1.wip.patch, yetus-local-testpatch-output-009.txt
>
>
> It is easily reproducible.
> client's hbase-site.xml: hadoop.security.authentication:kerberos, 
> hbase.security.authentication:kerberos, 
> hbase.ipc.client.fallback-to-simple-auth-allowed:true, keytab and principal 
> are right set
> A simple auth hbase cluster, a kerberized hbase client application. 
> application trying to r/w/c/d table will have following exception:
> {code}
> javax.security.sasl.SaslException: GSS initiate failed [Caused by 
> GSSException: No valid credentials provided (Mechanism level: Failed to find 
> any Kerberos tgt)]
>   at 
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
>   at 
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
>   at 
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
>   at 
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
>   at 
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:58383)
>   at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1592)
>   at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1530)
>   at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1552)
>   at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1581)
>   at 
>

UNCHECKED [jira] [Commented] (HBASE-21013) Backport "read part" of HBASE-18754 to all active 1.x branches

2018-09-19 Thread Mingdao Yang (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620436#comment-16620436
 ] 

Mingdao Yang commented on HBASE-21013:
--

Hi [~zghaobac],

I'm still working on it. I'm a newbie to Hbase.
I apologize for the delay. I'll provide an update soon.

Thanks.


> Backport "read part" of HBASE-18754 to all active 1.x branches
> --
>
> Key: HBASE-21013
> URL: https://issues.apache.org/jira/browse/HBASE-21013
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Mingdao Yang
>Priority: Critical
> Fix For: 1.5.0, 1.3.3, 1.2.8, 1.4.8
>
>
> The hfiles impacted by HBASE-18754 will have bytes of proto.TimeRangeTracker. 
> It makes all 1.x branches failed to read the hfile since all 1.x branches 
> can't deserialize the proto.TimeRangeTracker.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20993) [Auth] IPC client fallback to simple auth allowed doesn't work

2018-09-19 Thread Hudson (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620427#comment-16620427
 ] 

Hudson commented on HBASE-20993:


Results for branch branch-1
[build #466 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/466/]: 
(x) *{color:red}-1 overall{color}*

details (if available):

(x) {color:red}-1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/466//General_Nightly_Build_Report/]


(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/466//JDK7_Nightly_Build_Report/]


(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/466//JDK8_Nightly_Build_Report_(Hadoop2)/]




(x) {color:red}-1 source release artifact{color}
-- See build output for details.


> [Auth] IPC client fallback to simple auth allowed doesn't work
> --
>
> Key: HBASE-20993
> URL: https://issues.apache.org/jira/browse/HBASE-20993
> Project: HBase
>  Issue Type: Bug
>  Components: Client, IPC/RPC, security
>Affects Versions: 1.2.6, 1.3.2, 1.2.7, 1.4.7
>Reporter: Reid Chan
>Assignee: Jack Bearden
>Priority: Critical
> Fix For: 1.5.0, 1.4.8
>
> Attachments: HBASE-20993.001.patch, 
> HBASE-20993.003.branch-1.flowchart.png, HBASE-20993.branch-1.002.patch, 
> HBASE-20993.branch-1.003.patch, HBASE-20993.branch-1.004.patch, 
> HBASE-20993.branch-1.005.patch, HBASE-20993.branch-1.006.patch, 
> HBASE-20993.branch-1.007.patch, HBASE-20993.branch-1.008.patch, 
> HBASE-20993.branch-1.009.patch, HBASE-20993.branch-1.009.patch, 
> HBASE-20993.branch-1.2.001.patch, HBASE-20993.branch-1.wip.002.patch, 
> HBASE-20993.branch-1.wip.patch, yetus-local-testpatch-output-009.txt
>
>
> It is easily reproducible.
> client's hbase-site.xml: hadoop.security.authentication:kerberos, 
> hbase.security.authentication:kerberos, 
> hbase.ipc.client.fallback-to-simple-auth-allowed:true, keytab and principal 
> are right set
> A simple auth hbase cluster, a kerberized hbase client application. 
> application trying to r/w/c/d table will have following exception:
> {code}
> javax.security.sasl.SaslException: GSS initiate failed [Caused by 
> GSSException: No valid credentials provided (Mechanism level: Failed to find 
> any Kerberos tgt)]
>   at 
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
>   at 
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
>   at 
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
>   at 
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
>   at 
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:58383)
>   at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1592)
>   at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1530)
>   at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1552)
>   at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1581)
>   at 
>

[jira] [Commented] (HBASE-17190) eclipse compile master error

2018-09-19 Thread Alim (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-17190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620330#comment-16620330
 ] 

Alim commented on HBASE-17190:
--

Encountered the same issue while creating a new project using maven archetype 
plugin. While checking the parent pom ,found the plexus compiler version to be 
2.8.5(latest) and the maven compiler plugin version to be 3.1 (old) . When we 
invoked the plugin management block to invoke maven-compiler version 3.8.0 and 
surefire-plugin as 2.22.0(latest),the build went through .

This is a manual step but can we get this versions updated within parent pom 
just so we don't get this issue. 

> eclipse compile master error
> 
>
> Key: HBASE-17190
> URL: https://issues.apache.org/jira/browse/HBASE-17190
> Project: HBase
>  Issue Type: Bug
> Environment: windows 7 + eclipse + maven3.3.9
>Reporter: Jian Yi
>Priority: Minor
>
> [INFO] --- maven-compiler-plugin:3.2:compile (default-compile) @ hbase-thrift 
> ---
> [WARNING] The POM for org.apache.maven.shared:maven-shared-utils:jar:0.1 is 
> invalid, transitive dependencies (if any) will not be available, enable debug 
> logging for more details
> [WARNING] The POM for 
> org.apache.maven.shared:maven-shared-incremental:jar:1.1 is invalid, 
> transitive dependencies (if any) will not be available, enable debug logging 
> for more details
> [WARNING] The POM for org.codehaus.plexus:plexus-compiler-api:jar:2.4 is 
> invalid, transitive dependencies (if any) will not be available, enable debug 
> logging for more details
> [WARNING] The POM for org.codehaus.plexus:plexus-compiler-manager:jar:2.4 is 
> invalid, transitive dependencies (if any) will not be available, enable debug 
> logging for more details
> [WARNING] The POM for org.codehaus.plexus:plexus-compiler-javac:jar:2.4 is 
> invalid, transitive dependencies (if any) will not be available, enable debug 
> logging for more details
> [WARNING] Error injecting: org.apache.maven.plugin.compiler.CompilerMojo
> java.lang.NoClassDefFoundError: 
> org/codehaus/plexus/compiler/util/scan/mapping/SuffixMapping
>   at java.lang.Class.getDeclaredConstructors0(Native Method)
>   at java.lang.Class.privateGetDeclaredConstructors(Unknown Source)
>   at java.lang.Class.getDeclaredConstructors(Unknown Source)
>   at 
> com.google.inject.spi.InjectionPoint.forConstructorOf(InjectionPoint.java:245)
>   at 
> com.google.inject.internal.ConstructorBindingImpl.create(ConstructorBindingImpl.java:99)
>   at 
> com.google.inject.internal.InjectorImpl.createUninitializedBinding(InjectorImpl.java:658)
>   at 
> com.google.inject.internal.InjectorImpl.createJustInTimeBinding(InjectorImpl.java:882)
>   at 
> com.google.inject.internal.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:805)
>   at 
> com.google.inject.internal.InjectorImpl.getJustInTimeBinding(InjectorImpl.java:282)
>   at 
> com.google.inject.internal.InjectorImpl.getBindingOrThrow(InjectorImpl.java:214)
>   at 
> com.google.inject.internal.InjectorImpl.getProviderOrThrow(InjectorImpl.java:1006)
>   at 
> com.google.inject.internal.InjectorImpl.getProvider(InjectorImpl.java:1038)
>   at 
> com.google.inject.internal.InjectorImpl.getProvider(InjectorImpl.java:1001)
>   at 
> com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1051)
>   at 
> org.eclipse.sisu.space.AbstractDeferredClass.get(AbstractDeferredClass.java:48)
>   at 
> com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
>   at 
> com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53)
>   at 
> com.google.inject.internal.ProviderInternalFactory$1.call(ProviderInternalFactory.java:65)
>   at 
> com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:115)
>   at 
> org.eclipse.sisu.bean.BeanScheduler$Activator.onProvision(BeanScheduler.java:176)
>   at 
> com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126)
>   at 
> com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68)
>   at 
> com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:63)
>   at 
> com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45)
>   at 
> com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
>   at 
> com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
>   at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
>   at

[jira] [Commented] (HBASE-21206) Scan with batch size may return incomplete cells

2018-09-19 Thread Zheng Hu (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620290#comment-16620290
 ] 

Zheng Hu commented on HBASE-21206:
--

[~mdrob] , [~Apache9]   Any other concerns ?  If no, I'll prepare the patch for 
other branches, and push to git.  

> Scan with batch size may return incomplete cells
> 
>
> Key: HBASE-21206
> URL: https://issues.apache.org/jira/browse/HBASE-21206
> Project: HBase
>  Issue Type: Bug
>  Components: scan
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 1.4.8, 2.1.1, 2.0.3
>
> Attachments: HBASE-21206.v1.patch, HBASE-21206.v1.patch, 
> HBASE-21206.v2.patch, HBASE-21206.v3.patch, ut.patch
>
>
> See the attached UT.  the table has 5 columns and each column has at least 
> one cell in it, but when we scan the table with batchSize=3,  we only got 3 
> cells returned , the other 2 cells got lost ...
> It's a critial bug and should be fixed..



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21156) [hbck2] Queue an assign of hbase:meta and bulk assign/unassign

2018-09-19 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620284#comment-16620284
 ] 

Hadoop QA commented on HBASE-21156:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} branch-2.1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
13s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 
26s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
56s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  3m 
31s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
55s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
57s{color} | {color:green} branch-2.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
 2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 15m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m 
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
35s{color} | {color:red} hbase-client: The patch generated 4 new + 211 
unchanged - 0 fixed = 215 total (was 211) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m 
16s{color} | {color:red} hbase-server: The patch generated 2 new + 198 
unchanged - 2 fixed = 200 total (was 200) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  3m 
45s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}  
8m 42s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4 
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}  
1m 23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
30s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m  
9s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}135m 
55s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
57s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}215m 22s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:42ca976 |
| JIRA

UNCHECKED [jira] [Commented] (HBASE-20623) Introduce the helper method "getCellBuilder()" to Mutation

2018-09-19 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620279#comment-16620279
 ] 

Hadoop QA commented on HBASE-20623:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}  
0m  0s{color} | {color:orange} The patch doesn't appear to include any new or 
modified tests. Please justify why no new tests are needed for this patch. Also 
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
 9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
20s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
38s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
 6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
33s{color} | {color:red} hbase-client: The patch generated 3 new + 42 unchanged 
- 0 fixed = 45 total (was 42) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
21s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
11m 44s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
25s{color} | {color:red} hbase-client generated 1 new + 2 unchanged - 0 fixed = 
3 total (was 2) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
44s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
18s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
25s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 59s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20623 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12929561/HBASE-20623.master.002.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux c203f483e8c4 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27 
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |

[jira] [Commented] (HBASE-21023) Add bypassProcedureToCompletion() API to HbckService

2018-09-19 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620276#comment-16620276
 ] 

Hadoop QA commented on HBASE-21023:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} branch-2.1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m  
2s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
43s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 
35s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
 9s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  3m 
31s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m 
32s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
17s{color} | {color:green} branch-2.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
 3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 15m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m 
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
31s{color} | {color:red} hbase-client: The patch generated 1 new + 0 unchanged 
- 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
14s{color} | {color:red} hbase-procedure: The patch generated 2 new + 10 
unchanged - 0 fixed = 12 total (was 10) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m 
12s{color} | {color:red} hbase-server: The patch generated 2 new + 11 unchanged 
- 0 fixed = 13 total (was 11) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  3m 
26s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}  
8m 26s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4 
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}  
1m 40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  2m  
3s{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 
1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
12s{color} | {color:red} hbase-procedure generated 1 new + 0 unchanged - 0 
fixed = 1 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
31s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m  
3s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
51s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}135m 31s{color} 
| {color:red}

[jira] [Commented] (HBASE-21013) Backport "read part" of HBASE-18754 to all active 1.x branches

2018-09-19 Thread Guanghao Zhang (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620250#comment-16620250
 ] 

Guanghao Zhang commented on HBASE-21013:


Any updates here? As we plan to rolling update our 0.98 cluster to 2.1, we 
already have a internal patch for 0.98. If no updates here, we can contribute 
our patch for branch-1.x, too. Thanks.

> Backport "read part" of HBASE-18754 to all active 1.x branches
> --
>
> Key: HBASE-21013
> URL: https://issues.apache.org/jira/browse/HBASE-21013
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Mingdao Yang
>Priority: Critical
> Fix For: 1.5.0, 1.3.3, 1.2.8, 1.4.8
>
>
> The hfiles impacted by HBASE-18754 will have bytes of proto.TimeRangeTracker. 
> It makes all 1.x branches failed to read the hfile since all 1.x branches 
> can't deserialize the proto.TimeRangeTracker.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20623) Introduce the helper method "getCellBuilder()" to Mutation

2018-09-19 Thread Reid Chan (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-20623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620234#comment-16620234
 ] 

Reid Chan commented on HBASE-20623:
---

Bump, may i ask how's going on, is it still on working?

> Introduce the helper method "getCellBuilder()" to Mutation
> --
>
> Key: HBASE-20623
> URL: https://issues.apache.org/jira/browse/HBASE-20623
> Project: HBase
>  Issue Type: Task
>  Components: API
>Reporter: Chia-Ping Tsai
>Assignee: maoling
>Priority: Minor
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-20623.master.001.patch, 
> HBASE-20623.master.002.patch
>
>
> see 
> [https://lists.apache.org/thread.html/d05bfaa0134502a47f6e1aca56cb0b096d4dd32ddefbbdf28db4952a@%3Cdev.hbase.apache.org%3E]
>  for more details.
> {code:java}
> How about a "getCellBuilder" or "getCellBuilderFactory" method for
> Mutation implementations that gives you a CellBuilder instance that
> already has relevant parts set? Like for a Put instance it should be
> able to already have the Type and Row set.{code}
> mentioned a day or so ago by [~busbey]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21204) NPE when scan raw DELETE_FAMILY_VERSION and codec is not set

2018-09-19 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HBASE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620174#comment-16620174
 ] 

Hadoop QA commented on HBASE-21204:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
12s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m  
8s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
32s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  3m 
36s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  6m 
15s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 14m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m 
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m 
20s{color} | {color:red} hbase-server: The patch generated 1 new + 19 unchanged 
- 0 fixed = 20 total (was 19) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
13s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}  
9m 57s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4 
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}  
1m 44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  6m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
34s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
24s{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}127m 38s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
56s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}210m 41s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestRegionServerAbort |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:42ca976 |
| JIRA Issue | HBASE-21204 |
| JIRA Patch URL |

59 matches

Mail list logo