[jira] [Commented] (HBASE-20531) RS may throw NPE when close meta regions in shutdown procedure.

2018-05-06 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-20531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465115#comment-16465115
 ] 

Hudson commented on HBASE-20531:


Results for branch master
[build #322 on 
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/322/]: (x) 
*{color:red}-1 overall{color}*

details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general 
report|https://builds.apache.org/job/HBase%20Nightly/job/master/322//General_Nightly_Build_Report/]




(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) 
report|https://builds.apache.org/job/HBase%20Nightly/job/master/322//JDK8_Nightly_Build_Report_(Hadoop2)/]


(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) 
report|https://builds.apache.org/job/HBase%20Nightly/job/master/322//JDK8_Nightly_Build_Report_(Hadoop3)/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


> RS may throw NPE when close meta regions in shutdown procedure. 
> 
>
> Key: HBASE-20531
> URL: https://issues.apache.org/jira/browse/HBASE-20531
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20531.v1.patch
>
>
> See also : 
> https://issues.apache.org/jira/browse/HBASE-20475?focusedCommentId=16463322&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16463322
> The NPE stack is as following: 
> {code}
> 2018-05-03 21:05:58,075 ERROR [RS_CLOSE_REGION-regionserver/instance-2:0-1] 
> helpers.MarkerIgnoringBase(159): * ABORTING region server 
> instance-2.c.gcp-hbase.internal,42063,1525381545380: Unrecoverable exception 
> while closing region tes
> t,,1525381436038.66de217a470764f3b37d8faebfd8e8c8., still finishing close 
> *
> java.io.IOException: java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1637)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1466)
> at 
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104)
> at 
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportFileArchivalForQuotas(HRegionServer.java:3709)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.reportArchivedFilesForQuota(HStore.java:2718)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.removeCompactedfiles(HStore.java:2649)
> at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:929)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1615)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1612)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> {code}
> In HRegionServer#run(),  we have the following: 
> {code}
> @Override
> public void run() {
> ..
> // Stop the quota manager
> if (rsQuotaManager != null) {
>   rsQuotaManager.stop();
> }
> if (rsSpaceQuotaManager != null) {
>   rsSpaceQuotaManager.stop();
>   rsSpaceQuotaManager = null;
> }
> ..
> // Closing the compactSplit thread before closing meta regions
> if (!this.killed && containsMetaTableRegions()) {
>   if (!abortRequested || this.fsOk) {
> if (this.compactSplitThread != null) {
>   this.compactSplitThread.join();
>   this.compactSplitThread = null;
> }
> closeMetaTableRegions(abortRequested);
>   }
> }
> ..
> }
> {code}
> We  stop the rsSpaceQuotaManager firstly, and then close the meta region, but 
> when close meta region, we need to use rsSpaceQuotaManager to 
> reportFileArchivalForQuotas() , just as the stack trace  said ... 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20531) RS may throw NPE when close meta regions in shutdown procedure.

2018-05-05 Thread Zheng Hu (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-20531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464974#comment-16464974
 ] 

Zheng Hu commented on HBASE-20531:
--

Check the branches,  the bug only existed in master branch, and I've pushed to 
master branch. Thanks [~yuzhih...@gmail.com] for reviewing. 

> RS may throw NPE when close meta regions in shutdown procedure. 
> 
>
> Key: HBASE-20531
> URL: https://issues.apache.org/jira/browse/HBASE-20531
> Project: HBase
>  Issue Type: Bug
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Attachments: HBASE-20531.v1.patch
>
>
> See also : 
> https://issues.apache.org/jira/browse/HBASE-20475?focusedCommentId=16463322&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16463322
> The NPE stack is as following: 
> {code}
> 2018-05-03 21:05:58,075 ERROR [RS_CLOSE_REGION-regionserver/instance-2:0-1] 
> helpers.MarkerIgnoringBase(159): * ABORTING region server 
> instance-2.c.gcp-hbase.internal,42063,1525381545380: Unrecoverable exception 
> while closing region tes
> t,,1525381436038.66de217a470764f3b37d8faebfd8e8c8., still finishing close 
> *
> java.io.IOException: java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1637)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1466)
> at 
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104)
> at 
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportFileArchivalForQuotas(HRegionServer.java:3709)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.reportArchivedFilesForQuota(HStore.java:2718)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.removeCompactedfiles(HStore.java:2649)
> at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:929)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1615)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1612)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> {code}
> In HRegionServer#run(),  we have the following: 
> {code}
> @Override
> public void run() {
> ..
> // Stop the quota manager
> if (rsQuotaManager != null) {
>   rsQuotaManager.stop();
> }
> if (rsSpaceQuotaManager != null) {
>   rsSpaceQuotaManager.stop();
>   rsSpaceQuotaManager = null;
> }
> ..
> // Closing the compactSplit thread before closing meta regions
> if (!this.killed && containsMetaTableRegions()) {
>   if (!abortRequested || this.fsOk) {
> if (this.compactSplitThread != null) {
>   this.compactSplitThread.join();
>   this.compactSplitThread = null;
> }
> closeMetaTableRegions(abortRequested);
>   }
> }
> ..
> }
> {code}
> We  stop the rsSpaceQuotaManager firstly, and then close the meta region, but 
> when close meta region, we need to use rsSpaceQuotaManager to 
> reportFileArchivalForQuotas() , just as the stack trace  said ... 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20531) RS may throw NPE when close meta regions in shutdown procedure.

2018-05-04 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-20531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464008#comment-16464008
 ] 

Hadoop QA commented on HBASE-20531:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
18s{color} | {color:green} branch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  4m 
17s{color} | {color:green} patch has no errors when building our shaded 
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
13m  7s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}166m 
22s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}207m 49s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20531 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12921936/HBASE-20531.v1.patch |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 0d971b3e0d9e 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11 
12:16:42 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 87f5b5f341 |
| maven | version: Apache Maven 3.5.3 
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC3 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/12723/testReport/ |
| Max. process+thread count | 4716 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/12723/console |
| P

[jira] [Commented] (HBASE-20531) RS may throw NPE when close meta regions in shutdown procedure.

2018-05-04 Thread Ted Yu (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-20531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463781#comment-16463781
 ] 

Ted Yu commented on HBASE-20531:


Lgtm

> RS may throw NPE when close meta regions in shutdown procedure. 
> 
>
> Key: HBASE-20531
> URL: https://issues.apache.org/jira/browse/HBASE-20531
> Project: HBase
>  Issue Type: Bug
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Attachments: HBASE-20531.v1.patch
>
>
> See also : 
> https://issues.apache.org/jira/browse/HBASE-20475?focusedCommentId=16463322&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16463322
> The NPE stack is as following: 
> {code}
> 2018-05-03 21:05:58,075 ERROR [RS_CLOSE_REGION-regionserver/instance-2:0-1] 
> helpers.MarkerIgnoringBase(159): * ABORTING region server 
> instance-2.c.gcp-hbase.internal,42063,1525381545380: Unrecoverable exception 
> while closing region tes
> t,,1525381436038.66de217a470764f3b37d8faebfd8e8c8., still finishing close 
> *
> java.io.IOException: java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1637)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1466)
> at 
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104)
> at 
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportFileArchivalForQuotas(HRegionServer.java:3709)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.reportArchivedFilesForQuota(HStore.java:2718)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.removeCompactedfiles(HStore.java:2649)
> at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:929)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1615)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1612)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> {code}
> In HRegionServer#run(),  we have the following: 
> {code}
> @Override
> public void run() {
> ..
> // Stop the quota manager
> if (rsQuotaManager != null) {
>   rsQuotaManager.stop();
> }
> if (rsSpaceQuotaManager != null) {
>   rsSpaceQuotaManager.stop();
>   rsSpaceQuotaManager = null;
> }
> ..
> // Closing the compactSplit thread before closing meta regions
> if (!this.killed && containsMetaTableRegions()) {
>   if (!abortRequested || this.fsOk) {
> if (this.compactSplitThread != null) {
>   this.compactSplitThread.join();
>   this.compactSplitThread = null;
> }
> closeMetaTableRegions(abortRequested);
>   }
> }
> ..
> }
> {code}
> We  stop the rsSpaceQuotaManager firstly, and then close the meta region, but 
> when close meta region, we need to use rsSpaceQuotaManager to 
> reportFileArchivalForQuotas() , just as the stack trace  said ... 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HBASE-20531) RS may throw NPE when close meta regions in shutdown procedure.

2018-05-04 Thread Zheng Hu (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-20531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463683#comment-16463683
 ] 

Zheng Hu commented on HBASE-20531:
--

The patch will be quite easy,  adjust the order of  stopping 
rsSpaceQuotaManager and closeMetaTableRegions ,  and check the null pointer in 
reportFileArchivalForQuotas ().

> RS may throw NPE when close meta regions in shutdown procedure. 
> 
>
> Key: HBASE-20531
> URL: https://issues.apache.org/jira/browse/HBASE-20531
> Project: HBase
>  Issue Type: Bug
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
>
> See also : 
> https://issues.apache.org/jira/browse/HBASE-20475?focusedCommentId=16463322&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16463322
> The NPE stack is as following: 
> {code}
> 2018-05-03 21:05:58,075 ERROR [RS_CLOSE_REGION-regionserver/instance-2:0-1] 
> helpers.MarkerIgnoringBase(159): * ABORTING region server 
> instance-2.c.gcp-hbase.internal,42063,1525381545380: Unrecoverable exception 
> while closing region tes
> t,,1525381436038.66de217a470764f3b37d8faebfd8e8c8., still finishing close 
> *
> java.io.IOException: java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1637)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1466)
> at 
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104)
> at 
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportFileArchivalForQuotas(HRegionServer.java:3709)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.reportArchivedFilesForQuota(HStore.java:2718)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.removeCompactedfiles(HStore.java:2649)
> at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:929)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1615)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1612)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> {code}
> In HRegionServer#run(),  we have the following: 
> {code}
> @Override
> public void run() {
> ..
> // Stop the quota manager
> if (rsQuotaManager != null) {
>   rsQuotaManager.stop();
> }
> if (rsSpaceQuotaManager != null) {
>   rsSpaceQuotaManager.stop();
>   rsSpaceQuotaManager = null;
> }
> ..
> // Closing the compactSplit thread before closing meta regions
> if (!this.killed && containsMetaTableRegions()) {
>   if (!abortRequested || this.fsOk) {
> if (this.compactSplitThread != null) {
>   this.compactSplitThread.join();
>   this.compactSplitThread = null;
> }
> closeMetaTableRegions(abortRequested);
>   }
> }
> ..
> }
> {code}
> We  stop the rsSpaceQuotaManager firstly, and then close the meta region, but 
> when close meta region, we need to use rsSpaceQuotaManager to 
> reportFileArchivalForQuotas() , just as the stack trace  said ... 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)