[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226293#comment-16226293 ] Anoop Sam John commented on HBASE-18375: bq.FAILURE: Integrated in Jenkins build HBase-2.0 #467 (See https://builds.apache.org/job/HBase-2.0/467/) Ya seems this is already there in branch-2. Around 7th Sep. So which version? Alpha-3? [~stack]? > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02-branch-2.patch, > HBASE-18375-V02.patch, HBASE-18375-V03.patch, HBASE-18375-V04.patch, > HBASE-18375-V05.patch, HBASE-18375-V06.patch, HBASE-18375-V07.patch, > HBASE-18375-V08.patch, HBASE-18375-V09.patch, HBASE-18375-V10.patch, > HBASE-18375-V11.patch, HBASE-18375-branch-2.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16156671#comment-16156671 ] Anastasia Braginsky commented on HBASE-18375: - When I pushed it to branch-2 I got some warning message from git... So I wasn't sure, if it was committed... Now looks like it was... > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-branch-2.patch, HBASE-18375-V01.patch, > HBASE-18375-V02-branch-2.patch, HBASE-18375-V02.patch, HBASE-18375-V03.patch, > HBASE-18375-V04.patch, HBASE-18375-V05.patch, HBASE-18375-V06.patch, > HBASE-18375-V07.patch, HBASE-18375-V08.patch, HBASE-18375-V09.patch, > HBASE-18375-V10.patch, HBASE-18375-V11.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16156446#comment-16156446 ] Anoop Sam John commented on HBASE-18375: Not yet pushed to 2.0? > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-branch-2.patch, HBASE-18375-V01.patch, > HBASE-18375-V02-branch-2.patch, HBASE-18375-V02.patch, HBASE-18375-V03.patch, > HBASE-18375-V04.patch, HBASE-18375-V05.patch, HBASE-18375-V06.patch, > HBASE-18375-V07.patch, HBASE-18375-V08.patch, HBASE-18375-V09.patch, > HBASE-18375-V10.patch, HBASE-18375-V11.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16155953#comment-16155953 ] Hudson commented on HBASE-18375: FAILURE: Integrated in Jenkins build HBase-2.0 #467 (See [https://builds.apache.org/job/HBase-2.0/467/]) HBASE-18375: Fix the bug where the pool chunks from ChunkCreator are (anastas: rev 68ec2a9da022f7824e9a45ef89a0c4b8bcb838f3) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ChunkCreator.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactingMemStore.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemstoreLABWithoutPool.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionPipeline.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreLABImpl.java > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-branch-2.patch, HBASE-18375-V01.patch, > HBASE-18375-V02-branch-2.patch, HBASE-18375-V02.patch, HBASE-18375-V03.patch, > HBASE-18375-V04.patch, HBASE-18375-V05.patch, HBASE-18375-V06.patch, > HBASE-18375-V07.patch, HBASE-18375-V08.patch, HBASE-18375-V09.patch, > HBASE-18375-V10.patch, HBASE-18375-V11.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16155578#comment-16155578 ] Anastasia Braginsky commented on HBASE-18375: - [~apurtell], are there pushes into branch-2 still possible? > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-branch-2.patch, HBASE-18375-V01.patch, > HBASE-18375-V02-branch-2.patch, HBASE-18375-V02.patch, HBASE-18375-V03.patch, > HBASE-18375-V04.patch, HBASE-18375-V05.patch, HBASE-18375-V06.patch, > HBASE-18375-V07.patch, HBASE-18375-V08.patch, HBASE-18375-V09.patch, > HBASE-18375-V10.patch, HBASE-18375-V11.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[
https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16151847#comment-16151847
]
Hadoop QA commented on HBASE-18375:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
12s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
34m 19s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 94m
30s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}143m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:3c8b364 |
| JIRA Issue | HBASE-18375 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12885135/HBASE-18375-branch-2.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux c0938e09d239 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-2 / a37417c |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8448/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/8448/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> The pool chunks from ChunkCreator are deallocated while in pool because there
> is no reference to them
> -
>
> Key: HBASE-18375
> URL: https://issues.apache.org/jira/browse/HBASE-18375
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0-alpha-1
>Reporter: Anastas
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16151815#comment-16151815 ] Anastasia Braginsky commented on HBASE-18375: - Attaching a freshly rebased patch for branch-2... > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-branch-2.patch, HBASE-18375-V01.patch, > HBASE-18375-V02-branch-2.patch, HBASE-18375-V02.patch, HBASE-18375-V03.patch, > HBASE-18375-V04.patch, HBASE-18375-V05.patch, HBASE-18375-V06.patch, > HBASE-18375-V07.patch, HBASE-18375-V08.patch, HBASE-18375-V09.patch, > HBASE-18375-V10.patch, HBASE-18375-V11.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[
https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16143052#comment-16143052
]
Hadoop QA commented on HBASE-18375:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
33s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 27s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 88m
0s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}133m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:3c8b364 |
| JIRA Issue | HBASE-18375 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12883966/HBASE-18375-branch-2.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 39aa2a167465 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-2 / 6db3d81 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8346/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/8346/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> The pool chunks from ChunkCreator are deallocated while in pool because there
> is no reference to them
> -
>
> Key: HBASE-18375
> URL: https://issues.apache.org/jira/browse/HBASE-18375
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0-alpha-1
>Reporter: Anastas
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16143026#comment-16143026 ] Anastasia Braginsky commented on HBASE-18375: - As the amount of unrelated failed tests is surprising, I have rebased and submitted once again > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-branch-2.patch, HBASE-18375-V01.patch, > HBASE-18375-V02-branch-2.patch, HBASE-18375-V02.patch, HBASE-18375-V03.patch, > HBASE-18375-V04.patch, HBASE-18375-V05.patch, HBASE-18375-V06.patch, > HBASE-18375-V07.patch, HBASE-18375-V08.patch, HBASE-18375-V09.patch, > HBASE-18375-V10.patch, HBASE-18375-V11.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[
https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16142752#comment-16142752
]
Hadoop QA commented on HBASE-18375:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
48s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
19s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
35m 15s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 54s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}135m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.master.procedure.TestDisableTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestCreateTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestModifyTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestEnableTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestDeleteTableProcedure |
| | org.apache.hadoop.hbase.client.TestSnapshotCloneIndependence |
| | org.apache.hadoop.hbase.master.TestGetLastFlushedSequenceId |
| | org.apache.hadoop.hbase.client.TestFromClientSide |
| | org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancer2 |
| | org.apache.hadoop.hbase.namespace.TestNamespaceAuditor |
| | org.apache.hadoop.hbase.TestHBaseTestingUtility |
| | org.apache.hadoop.hbase.mapred.TestTableInputFormat |
| | org.apache.hadoop.hbase.client.TestAsyncNamespaceAdminApi |
| | org.apache.hadoop.hbase.client.TestAsyncProcedureAdminApi |
| | org.apache.hadoop.hbase.TestMultiVersions |
| | org.apache.hadoop.hbase.client.TestAsyncNonMetaRegionLocator |
| | org.apache.hadoop.hbase.client.TestAsyncSnapshotAdminApi |
| | org.apache.hadoop.hbase.security.access.TestAccessController2 |
| | org.apache.hadoop.hbase.mapred.TestTableMapReduceUtil |
| | org.apache.hadoop.hbase.quotas.TestMasterSpaceQuotaObserver |
| | org.apache.hadoop.hbase.client.TestAsyncRegionLocatorTimeout |
| | org.apache.hadoop.hbase.TestZooKeeper |
| | org
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16142693#comment-16142693 ] Anastasia Braginsky commented on HBASE-18375: - Thank you, [~apurtell]! Did your suggestion of deleting and submitting the patch agin. > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-branch-2.patch, HBASE-18375-V01.patch, > HBASE-18375-V02-branch-2.patch, HBASE-18375-V02.patch, HBASE-18375-V03.patch, > HBASE-18375-V04.patch, HBASE-18375-V05.patch, HBASE-18375-V06.patch, > HBASE-18375-V07.patch, HBASE-18375-V08.patch, HBASE-18375-V09.patch, > HBASE-18375-V10.patch, HBASE-18375-V11.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16140311#comment-16140311 ] Andrew Purtell commented on HBASE-18375: [~anastas] This is a hacky suggestion, but if 'HBASE-18375-branch-2.patch' worked earlier, use it again. Attach the new patch using the same name. JIRA sorts patches by most recent so as long as hadoopqa does the right thing by choosing the first in the list, it will work. As a last resort, if you think hadoopqa isn't picking up the latest (although I believe it does), delete the previously attached branch-2 patches and then attach only the latest. > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-branch-2.patch, HBASE-18375-V01.patch, > HBASE-18375-V02-branch-2.patch, HBASE-18375-V02.patch, HBASE-18375-V03.patch, > HBASE-18375-V04.patch, HBASE-18375-V05.patch, HBASE-18375-V06.patch, > HBASE-18375-V07.patch, HBASE-18375-V08.patch, HBASE-18375-V09.patch, > HBASE-18375-V10.patch, HBASE-18375-V11.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16140176#comment-16140176 ] Anastasia Braginsky commented on HBASE-18375: - Hi [~anoop.hbase], [~ram_krish], [~stack], [~apurtell], I need some help here. I am trying to promote the fix for Release 2.0, so the MSLAB chunks there are returned back to pool. I have submitted a patch (relative to branch-2 with required name HBASE-18375-branch-2.patch) which failed QA. I then fixed the problem but unable to submit the new patch for QA. When I am submitting it once again under same name, the Hadoop QA process doesn't happen. When I submit under name HBASE-18375-V02-branch-2.patch QA tries to apply the patch on master and that fails... Any suggestions? Thanks! > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-branch-2.patch, HBASE-18375-V01.patch, > HBASE-18375-V02-branch-2.patch, HBASE-18375-V02.patch, HBASE-18375-V03.patch, > HBASE-18375-V04.patch, HBASE-18375-V05.patch, HBASE-18375-V06.patch, > HBASE-18375-V07.patch, HBASE-18375-V08.patch, HBASE-18375-V09.patch, > HBASE-18375-V10.patch, HBASE-18375-V11.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[
https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16138013#comment-16138013
]
Hadoop QA commented on HBASE-18375:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HBASE-18375 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.4.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-18375 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12883275/HBASE-18375-V02-branch-2.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/8261/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> The pool chunks from ChunkCreator are deallocated while in pool because there
> is no reference to them
> -
>
> Key: HBASE-18375
> URL: https://issues.apache.org/jira/browse/HBASE-18375
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0-alpha-1
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-18375-branch-2.patch, HBASE-18375-V01.patch,
> HBASE-18375-V02-branch-2.patch, HBASE-18375-V02.patch, HBASE-18375-V03.patch,
> HBASE-18375-V04.patch, HBASE-18375-V05.patch, HBASE-18375-V06.patch,
> HBASE-18375-V07.patch, HBASE-18375-V08.patch, HBASE-18375-V09.patch,
> HBASE-18375-V10.patch, HBASE-18375-V11.patch
>
>
> Because MSLAB list of chunks was changed to list of chunk IDs, the chunks
> returned back to pool can be deallocated by JVM because there is no reference
> to them. The solution is to protect pool chunks from GC by the strong map of
> ChunkCreator introduced by HBASE-18010. Will prepare the patch today.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[
https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16135271#comment-16135271
]
Hadoop QA commented on HBASE-18375:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
28s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
7s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m
46s{color} | {color:red} The patch causes 16 errors with Hadoop v2.4.0. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m
32s{color} | {color:red} The patch causes 16 errors with Hadoop v2.4.1. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m
19s{color} | {color:red} The patch causes 16 errors with Hadoop v2.5.0. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m
7s{color} | {color:red} The patch causes 16 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m
53s{color} | {color:red} The patch causes 16 errors with Hadoop v2.5.2. {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 21m 29s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestMemStoreLAB |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:3c8b364 |
| JIRA Issue | HBASE-18375 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882878/HBASE-18375-branch-2.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 1085c94b390e 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-2 / 95e1fa3 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC3 |
| u
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16135198#comment-16135198 ] Anastasia Braginsky commented on HBASE-18375: - Attached... > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-branch-2.patch, HBASE-18375-V01.patch, > HBASE-18375-V02.patch, HBASE-18375-V03.patch, HBASE-18375-V04.patch, > HBASE-18375-V05.patch, HBASE-18375-V06.patch, HBASE-18375-V07.patch, > HBASE-18375-V08.patch, HBASE-18375-V09.patch, HBASE-18375-V10.patch, > HBASE-18375-V11.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16135140#comment-16135140 ] Anoop Sam John commented on HBASE-18375: Can u attach the patch here with proper name convention for the patch? (HBASE-18375-branch-2.patch) > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch, HBASE-18375-V05.patch, > HBASE-18375-V06.patch, HBASE-18375-V07.patch, HBASE-18375-V08.patch, > HBASE-18375-V09.patch, HBASE-18375-V10.patch, HBASE-18375-V11.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16135042#comment-16135042 ] Anastasia Braginsky commented on HBASE-18375: - Hey, I have a patch for branch-2. The patch is very similar, but it is not identical to the patch I applied to master. What should I do so Hadoop QA runs over it? >From my experience, the tests I run on my machine and Hadoop QA often show >different results > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch, HBASE-18375-V05.patch, > HBASE-18375-V06.patch, HBASE-18375-V07.patch, HBASE-18375-V08.patch, > HBASE-18375-V09.patch, HBASE-18375-V10.patch, HBASE-18375-V11.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16134801#comment-16134801 ] Anastasia Braginsky commented on HBASE-18375: - Yes, pushed only to master. Will push to branch-2 as well. > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch, HBASE-18375-V05.patch, > HBASE-18375-V06.patch, HBASE-18375-V07.patch, HBASE-18375-V08.patch, > HBASE-18375-V09.patch, HBASE-18375-V10.patch, HBASE-18375-V11.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16134698#comment-16134698 ] Anoop Sam John commented on HBASE-18375: You mean you pushed patch to master branch alone? We need this in 2.0 also. Pls push to branch-2 > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch, HBASE-18375-V05.patch, > HBASE-18375-V06.patch, HBASE-18375-V07.patch, HBASE-18375-V08.patch, > HBASE-18375-V09.patch, HBASE-18375-V10.patch, HBASE-18375-V11.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16134016#comment-16134016 ] Anastasia Braginsky commented on HBASE-18375: - [~anoop.hbase] and [~ram_krish] thank you for your help and reviews! Anoop, Ram and [~stack], what about committing to Release 2.0 as well? If it is needed, what branch should be used? In Release 2.0 the MSLAB chunks may not return to the pool, but with low probability of this to happen. > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch, HBASE-18375-V05.patch, > HBASE-18375-V06.patch, HBASE-18375-V07.patch, HBASE-18375-V08.patch, > HBASE-18375-V09.patch, HBASE-18375-V10.patch, HBASE-18375-V11.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131163#comment-16131163 ] Hudson commented on HBASE-18375: FAILURE: Integrated in Jenkins build HBASE-14070.HLC #233 (See [https://builds.apache.org/job/HBASE-14070.HLC/233/]) HBASE-18375: Fix the bug where the pool chunks from ChunkCreator are (anastas: rev 75a6b36849c58d6a751f57226ab0c8f7884a9e87) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionPipeline.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemstoreLABWithoutPool.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CellChunkImmutableSegment.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreLABImpl.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ChunkCreator.java > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch, HBASE-18375-V05.patch, > HBASE-18375-V06.patch, HBASE-18375-V07.patch, HBASE-18375-V08.patch, > HBASE-18375-V09.patch, HBASE-18375-V10.patch, HBASE-18375-V11.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130648#comment-16130648 ] Anastasia Braginsky commented on HBASE-18375: - OK, committing... > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch, HBASE-18375-V05.patch, > HBASE-18375-V06.patch, HBASE-18375-V07.patch, HBASE-18375-V08.patch, > HBASE-18375-V09.patch, HBASE-18375-V10.patch, HBASE-18375-V11.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[
https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130642#comment-16130642
]
Hadoop QA commented on HBASE-18375:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
34s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 9s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}139m
38s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}187m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-18375 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882329/HBASE-18375-V11.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 822cfe6c0501 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 092dc6d |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8132/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/8132/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> The pool chunks from ChunkCreator are deallocated while in pool because there
> is no reference to them
> -
>
> Key: HBASE-18375
> URL: https://issues.apache.org/jira/browse/HBASE-18375
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0-alpha-1
>Reporter: Anastasia Bragins
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130305#comment-16130305 ] Anastasia Braginsky commented on HBASE-18375: - Have added the findbug suppressing comment, then got all those unrelated failures. So I have rebased and submitted another patch. > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch, HBASE-18375-V05.patch, > HBASE-18375-V06.patch, HBASE-18375-V07.patch, HBASE-18375-V08.patch, > HBASE-18375-V09.patch, HBASE-18375-V10.patch, HBASE-18375-V11.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[
https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130230#comment-16130230
]
Hadoop QA commented on HBASE-18375:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
36m 58s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 13s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}151m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.master.TestMasterMetricsWrapper |
| | org.apache.hadoop.hbase.master.procedure.TestDisableTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestModifyTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestCreateTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestEnableTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestDeleteTableProcedure |
| | org.apache.hadoop.hbase.client.TestSnapshotCloneIndependence |
| | org.apache.hadoop.hbase.master.TestGetLastFlushedSequenceId |
| | org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancer2 |
| | org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics |
| | org.apache.hadoop.hbase.namespace.TestNamespaceAuditor |
| | org.apache.hadoop.hbase.TestHBaseTestingUtility |
| | org.apache.hadoop.hbase.mapred.TestTableInputFormat |
| | org.apache.hadoop.hbase.master.TestMasterFileSystemWithWALDir |
| | org.apache.hadoop.hbase.TestMultiVersions |
| | org.apache.hadoop.hbase.quotas.TestQuotaStatusRPCs |
| |
org.apache.hadoop.hbase.security.visibility.TestVisibilityLablesWithGroups |
| | org.apache.hadoop.hbase.client.TestAsyncNonMetaRegionLocator |
| | org.apache.hadoop.hbase.security.access.TestAccessController2 |
| | org.apache.hadoop.hbase.mapred.TestTableMapReduceUtil |
| | org.apache.hadoop.hbase.quotas.TestMasterSpaceQuotaObserver |
| | org.apache.hadoop.hbase.master.TestGetIn
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[
https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129987#comment-16129987
]
ramkrishna.s.vasudevan commented on HBASE-18375:
I think we should add some Findbug Tags to avoid this warning.
For eg
{code}
@edu.umd.cs.findbugs.annotations.SuppressWarnings(value="SF_SWITCH_FALLTHROUGH",
justification="Intentional")
{code}
Just try adding a warning and submit QA if you feel the findbug warning is not
needed to be fixed.
> The pool chunks from ChunkCreator are deallocated while in pool because there
> is no reference to them
> -
>
> Key: HBASE-18375
> URL: https://issues.apache.org/jira/browse/HBASE-18375
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0-alpha-1
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch,
> HBASE-18375-V03.patch, HBASE-18375-V04.patch, HBASE-18375-V05.patch,
> HBASE-18375-V06.patch, HBASE-18375-V07.patch, HBASE-18375-V08.patch,
> HBASE-18375-V09.patch
>
>
> Because MSLAB list of chunks was changed to list of chunk IDs, the chunks
> returned back to pool can be deallocated by JVM because there is no reference
> to them. The solution is to protect pool chunks from GC by the strong map of
> ChunkCreator introduced by HBASE-18010. Will prepare the patch today.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129978#comment-16129978 ] Anastasia Braginsky commented on HBASE-18375: - What do you think ? :) > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch, HBASE-18375-V05.patch, > HBASE-18375-V06.patch, HBASE-18375-V07.patch, HBASE-18375-V08.patch, > HBASE-18375-V09.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129977#comment-16129977 ] Anastasia Braginsky commented on HBASE-18375: - I saw the findbug. It is about volatile field "version" being increased in method replaceAtIndex() which doesn't have a synchronized block. However, the only invocation of replaceAtIndex() method is from synchronized block inside flattenOneSegment() method. So, after all the volatile field "version" is updated within synchronized block, which is stated in the comments in the code. It looks like the way to satisfy the "findbug" is to move the content of replaceAtIndex() method into flattenOneSegment() method, and this I do not like. Also to relay on re-entrance of synchronized method is bad here, because we must have synchronized block earlier and someone may think that this is enough. So I decided to leave it like this. [~ram_krish], hat do you think? > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch, HBASE-18375-V05.patch, > HBASE-18375-V06.patch, HBASE-18375-V07.patch, HBASE-18375-V08.patch, > HBASE-18375-V09.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129907#comment-16129907 ] ramkrishna.s.vasudevan commented on HBASE-18375: There is a findbug. Pls check it. > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch, HBASE-18375-V05.patch, > HBASE-18375-V06.patch, HBASE-18375-V07.patch, HBASE-18375-V08.patch, > HBASE-18375-V09.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[
https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129025#comment-16129025
]
Hadoop QA commented on HBASE-18375:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 3m
49s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 30s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
24s{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed =
1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}116m 54s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}168m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Increment of volatile field
org.apache.hadoop.hbase.regionserver.CompactionPipeline.version in
org.apache.hadoop.hbase.regionserver.CompactionPipeline.replaceAtIndex(int,
ImmutableSegment) At CompactionPipeline.java:in
org.apache.hadoop.hbase.regionserver.CompactionPipeline.replaceAtIndex(int,
ImmutableSegment) At CompactionPipeline.java:[line 288] |
| Timed out junit tests |
org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-18375 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882131/HBASE-18375-V09.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux a206256e577a 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 2a9cdd5 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC3 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/8118/artifact/patchprocess/new-findbugs-hbase-server.html
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/8118/artifact/patchprocess/patch-unit-hbase-
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128776#comment-16128776 ] Anastasia Braginsky commented on HBASE-18375: - The interesting point is that the difference between the previous patch (no test failures) and the current patch (lots of failures) is in a comment... :) Anyway, will try to rebase and submit once again > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch, HBASE-18375-V05.patch, > HBASE-18375-V06.patch, HBASE-18375-V07.patch, HBASE-18375-V08.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[
https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128731#comment-16128731
]
Hadoop QA commented on HBASE-18375:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 28s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
50s{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed =
1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 56s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}129m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Increment of volatile field
org.apache.hadoop.hbase.regionserver.CompactionPipeline.version in
org.apache.hadoop.hbase.regionserver.CompactionPipeline.replaceAtIndex(int,
ImmutableSegment) At CompactionPipeline.java:in
org.apache.hadoop.hbase.regionserver.CompactionPipeline.replaceAtIndex(int,
ImmutableSegment) At CompactionPipeline.java:[line 288] |
| Timed out junit tests |
org.apache.hadoop.hbase.client.TestSnapshotCloneIndependence |
| | org.apache.hadoop.hbase.snapshot.TestSnapshotClientRetries |
| | org.apache.hadoop.hbase.coprocessor.TestRegionObserverScannerOpenHook |
| | org.apache.hadoop.hbase.client.TestAsyncProcedureAdminApi |
| | org.apache.hadoop.hbase.TestFullLogReconstruction |
| | org.apache.hadoop.hbase.TestMultiVersions |
| | org.apache.hadoop.hbase.client.TestAsyncSnapshotAdminApi |
| | org.apache.hadoop.hbase.client.TestAsyncNonMetaRegionLocator |
| |
org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint
|
| | org.apache.hadoop.hbase.wal.TestWALFiltering |
| | org.apache.hadoop.hbase.client.TestAsyncClusterAdminApi |
| | org.apache.hadoop.hbase.quotas.TestMasterSpaceQuotaObserver |
| | org.apache.hadoop.hbase.TestZooKeeper |
| | org.apache.hadoop.hbase.client.TestAsyncDrainAdminApi |
| | org.apache.hadoop.hbase.client.TestAsyncTableAdminApi |
| | org.apache.hadoop.hbase.replication.regionserv
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128606#comment-16128606 ] Anastasia Braginsky commented on HBASE-18375: - New patch includes the TODO comment request by Ram. Thanks! > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch, HBASE-18375-V05.patch, > HBASE-18375-V06.patch, HBASE-18375-V07.patch, HBASE-18375-V08.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128438#comment-16128438 ] Anastasia Braginsky commented on HBASE-18375: - Hi Guys! As I am not getting any other feedback I am planning to commit this change by the EOD today. Please raise your objections if any. Thanks! > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch, HBASE-18375-V05.patch, > HBASE-18375-V06.patch, HBASE-18375-V07.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[
https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16125624#comment-16125624
]
Hadoop QA commented on HBASE-18375:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 57s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
26s{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed =
1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}108m
38s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}155m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Increment of volatile field
org.apache.hadoop.hbase.regionserver.CompactionPipeline.version in
org.apache.hadoop.hbase.regionserver.CompactionPipeline.replaceAtIndex(int,
ImmutableSegment) At CompactionPipeline.java:in
org.apache.hadoop.hbase.regionserver.CompactionPipeline.replaceAtIndex(int,
ImmutableSegment) At CompactionPipeline.java:[line 288] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-18375 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881706/HBASE-18375-V07.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 95f2ffe956ac 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / e2b797b |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC3 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/8072/artifact/patchprocess/new-findbugs-hbase-server.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8072/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HB
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16125493#comment-16125493 ] Anastasia Braginsky commented on HBASE-18375: - Hey! Thanks for the review! I have answered your comments on the review board. Please feel free to answer here or there. Publishing the recent patch, last I hope :) I would like to commit. What do you think? > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch, HBASE-18375-V05.patch, > HBASE-18375-V06.patch, HBASE-18375-V07.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[
https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16124953#comment-16124953
]
Hadoop QA commented on HBASE-18375:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 14s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
11s{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed =
1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}112m
41s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}158m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Increment of volatile field
org.apache.hadoop.hbase.regionserver.CompactionPipeline.version in
org.apache.hadoop.hbase.regionserver.CompactionPipeline.replaceAtIndex(int,
ImmutableSegment) At CompactionPipeline.java:in
org.apache.hadoop.hbase.regionserver.CompactionPipeline.replaceAtIndex(int,
ImmutableSegment) At CompactionPipeline.java:[line 284] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-18375 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881651/HBASE-18375-V06.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux dd7599934a8c 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 173dce7 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC3 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/8061/artifact/patchprocess/new-findbugs-hbase-server.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8061/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBA
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16124905#comment-16124905 ] Anastasia Braginsky commented on HBASE-18375: - Hey! :) Publishing one more patch (here and on RB) with code review comments fixes so far. Thank you [~eshcar] and [~anoop.hbase] !!! [~ram_krish], I'll wait to your review a bit, but I would like to finish with this JIRA asap and move forward. I have spent a bit too much time on this debugging... Thanks! > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch, HBASE-18375-V05.patch, > HBASE-18375-V06.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123722#comment-16123722 ] ramkrishna.s.vasudevan commented on HBASE-18375: Will have a look at this beginning of next week. Thanks. > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch, HBASE-18375-V05.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121672#comment-16121672 ] Eshcar Hillel commented on HBASE-18375: --- lgtm left some comments in review board +1 > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch, HBASE-18375-V05.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[
https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121610#comment-16121610
]
Hadoop QA commented on HBASE-18375:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
33m 14s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
17s{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed =
1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}143m
59s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}198m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Increment of volatile field
org.apache.hadoop.hbase.regionserver.CompactionPipeline.version in
org.apache.hadoop.hbase.regionserver.CompactionPipeline.replaceAtIndex(int,
ImmutableSegment) At CompactionPipeline.java:in
org.apache.hadoop.hbase.regionserver.CompactionPipeline.replaceAtIndex(int,
ImmutableSegment) At CompactionPipeline.java:[line 284] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-18375 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881182/HBASE-18375-V05.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 726361b056ad 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 6246523 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC3 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/8010/artifact/patchprocess/new-findbugs-hbase-server.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/8010/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreC
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121394#comment-16121394 ] Anastasia Braginsky commented on HBASE-18375: - [~anoop.hbase], [~ram_krish], [~eshcar], and everybody! I am publishing the patch where the non-pooled chunks are covered by stong-map only when MSLAB tells them to do so, and this based on whether this specific MSLAB is related to CompactingMemStore that works with CellChunkMap. This looks to me like an optimal solution takes into consideration everything. The patch is published here and on RB, please take a look and let me know what do you think. Thanks! > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch, HBASE-18375-V05.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120253#comment-16120253 ] Anoop Sam John commented on HBASE-18375: Yes I was also saying same. Pls see bq.You mean when the Active segment has used at least one Chunk NOT from pool, we wont flatten it into a CCM? So the all pooled chunks needed in that active segment itself which is becoming CCM. Will see ur patch. > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119743#comment-16119743 ] Anastasia Braginsky commented on HBASE-18375: - bq. You mean when the Active segment has used at least one Chunk NOT from pool, we wont flatten it into a CCM? No. I think I need to explain how CCM works. It is not that everything can work as usual and then only when flattening happens we can start managing the chunks. The record of chunks to their IDs must be done when chunk is created for active segment, meaning for CSLM. Later it is already to late, as we do not want to copy cells from one chunk to another. So if we say CCM is working only with pool chunks, also Active works only with pool. bq. Because we dont have Chunks in Pool, we can not make any writes to fail !! I agree with you. Therefore I believe the first option is better. > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119699#comment-16119699 ] Anoop Sam John commented on HBASE-18375: bq.2. Make CCM to work only with chunks from pool and throw exception when there are no more chunks in pool. The user will be requested to put all its memory to the pool in this case. You mean when the Active segment has used at least one Chunk NOT from pool, we wont flatten it into a CCM? May be there is a way to track this down? Because we dont have Chunks in Pool, we can not make any writes to fail !! > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119677#comment-16119677 ] Anastasia Braginsky commented on HBASE-18375: - bq. It is the ChunkMap impl which need to keep the ref to Chunks for longer. So I think it should be responsibility of that class to act. This is impossible, because chunks mapping need to be saved since the chunk was created and this is before any CellChunkMap exists. How common is the scenario of HBASE-16195 ? I see two options 1. StrongMap for pool and only when CCM is defined then for other chunks 2. Make CCM to work only with chunks from pool and throw exception when there are no more chunks in pool. The user will be requested to put all its memory to the pool in this case. I believe that the first option is much better. > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[
https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119635#comment-16119635
]
Anoop Sam John commented on HBASE-18375:
bq. If CellChunkMap is requested in hbase-site.xml
("hbase.hregion.compacting.memstore.index"), then StrongMap is going to cover
every chunk.
Otherwise, StrongMap is going to cover only chunks in pool.
Then there will be concern as in HBASE-16195.
It is the ChunkMap impl which need to keep the ref to Chunks for longer. So I
think it should be responsibility of that class to act. Also say we have used
10 chunks till now and then there is an in memory flush/ flatten to
CellChunkMap. Now may be some of the Cells become irrelevant and say only 5
Chunks have active refs from Cell. (Assume all these are Chunk not from pool)
So we can keep only 5 Chunks in the CellChunkImpl.
Need to see what to do where. But if we can do above way that is the best.
> The pool chunks from ChunkCreator are deallocated while in pool because there
> is no reference to them
> -
>
> Key: HBASE-18375
> URL: https://issues.apache.org/jira/browse/HBASE-18375
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0-alpha-1
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
>Priority: Critical
> Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch,
> HBASE-18375-V03.patch, HBASE-18375-V04.patch
>
>
> Because MSLAB list of chunks was changed to list of chunk IDs, the chunks
> returned back to pool can be deallocated by JVM because there is no reference
> to them. The solution is to protect pool chunks from GC by the strong map of
> ChunkCreator introduced by HBASE-18010. Will prepare the patch today.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[
https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119625#comment-16119625
]
Anastasia Braginsky commented on HBASE-18375:
-
[~ram_krish], when I wrote about ByteBuffers being freed we had slightly
different scenario, with more bugs (the one with version in pipeline not
increased upon in-memory flush), there everything was released including
ByteBuffers. After this was fixed, I continued to see the problem of chunks
being released unexpectedly. And now I see that the chunks are released, while
the ButeBuffers are not released.
My plan is as following:
1. To have *only* StrongMap in ChunkCreator
2. If CellChunkMap is requested in hbase-site.xml
("hbase.hregion.compacting.memstore.index"), then StrongMap is going to cover
every chunk.
Otherwise, StrongMap is going to cover only chunks in pool.
3. No change in MSLAB (chunkID list remains there).
I believe this change is applicable also to Release 2.0, just let me know what
branch to update. So I do not plan to do 2 JIRAs.
> The pool chunks from ChunkCreator are deallocated while in pool because there
> is no reference to them
> -
>
> Key: HBASE-18375
> URL: https://issues.apache.org/jira/browse/HBASE-18375
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0-alpha-1
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
>Priority: Critical
> Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch,
> HBASE-18375-V03.patch, HBASE-18375-V04.patch
>
>
> Because MSLAB list of chunks was changed to list of chunk IDs, the chunks
> returned back to pool can be deallocated by JVM because there is no reference
> to them. The solution is to protect pool chunks from GC by the strong map of
> ChunkCreator introduced by HBASE-18010. Will prepare the patch today.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118789#comment-16118789 ] Anoop Sam John commented on HBASE-18375: Thanks for the confirm. This is excellent investigation story! bq.As solution to Release 2.0 I would suggest to bring back the chunks (from pool) linked list into the MSLAB instead of chunk IDs linked list. This is for the pool related bug fix right. Ya that make sense. There is no point in NOT holding chunk refs in MSLAB (When from pool). Any way those chunks are not going to go off at any time. When CellChunkMap is in place, seems we can not release any chunk unless a CellChunkMap is been made (in memory flush flatten). This will have concern as said in HBASE-16195. So at least when the CellChunkMap is been made, we should remove the unreferred chunks. How u plan? Fix both issues here or in 2 jiras? > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118559#comment-16118559 ] ramkrishna.s.vasudevan commented on HBASE-18375: [~anastas] Thanks for clearing my doubt. In your earlier comment b. When S is closed, C is returned to ChunkCreator, which in turn returns C to the pool, but in parrallel the GC is already freeing C's "unreachable" ByteBuffer. You said the ByteBuffer were freed. So it is like the Chunks are cleared though there was an active ref to one of its member. Ya previously it was soft reference so the chance of that getting freed is lesser if we have enough memory. Thanks for this useful info. > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118358#comment-16118358 ] Anastasia Braginsky commented on HBASE-18375: - Hey [~anoop.hbase], Great minds thinks alike! We have just realized what the problem is here (in details) and I came to write the comment and saw your comment :) As [~ram_krish] has written we didn't know the true reason why this patch resolves the problem, so I continued to look for the "truth". So now we have the full understanding what is going on. Here it is: 1. The Maps in ChunkCreator are referencing chunks, while Cells in CSLM and CellArayMap are referencing ByteBuffers. 2. Therefore chunks that are referenced from WeakMap or SoftMap are getting released, however ByteBuffers are still there. 3. There is no meaning in WeakMap or SoftMap. Only StrongMap can truly save the chunks from GC while they are still in active CSLM. So only StrongMap really helps for CellChunkMap. >From here the patch presented here resolves a true problem of pool chunks >being not returned from pool. The problem exists also in Release 2.0 (!), but since the references in map there are soft references, which are released last, the frequency of problem to appear is less. As solution I suggest to use only StrongMap for all chunks. So let me present another patch for this JIRA. As solution to Release 2.0 I would suggest to bring back the chunks (from pool) linked list into the MSLAB instead of chunk IDs linked list. What do you think? bq. Are we always sure when the above is called, the weak map will have refs to Chunks still ? Clearly not, the chunks are released very fast, just after they were added to CSLM. > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[
https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118272#comment-16118272
]
Anoop Sam John commented on HBASE-18375:
One Q on this below code part
{code}
Chunk saveChunkFromGC(int chunkID) {
Chunk c = strongChunkIdMap.get(chunkID); // check whether the chunk is
already protected
if (c != null) // with strong pointer
return c;
WeakReference ref = weakChunkIdMap.get(chunkID);
if (ref != null) {
c = ref.get();
}
if (c != null) {
// put this strong reference to chunk into the strongChunkIdMap
// the read of the weakMap is always happening before the read of the
strongMap
// so no synchronization issues here
this.strongChunkIdMap.put(chunkID, c);
this.weakChunkIdMap.remove(chunkID);
return c;
}
// we should actually never return null as someone should not ask to save
from GC a chunk,
// which is already released. However, we are not asserting it here and we
let the caller
// to deal with the return value an assert if needed
return null;
}
{code}
My case is with not using pool. So when the chunk was created by the
ChunkCreator, the ref is put in weak map and no refs in strong map. So weak map
refer to the Chunk object as such. Now MSLAB impl class will have ref to
current Chunk (To which cells are written). Then we may move to next chunks
and old ones, we keep only chunk ids. The active Cells will refer to the
ByteBuffer within the Chunk. So no refs to Chunk object as such.
Are we always sure when the above is called, the weak map will have refs to
Chunks still ? The Chunk object as such is not referred from any other place
though the ByteBuffer is referred. Ya the Chunk is having ref to live BB
object but the Chunk object as such is unreferred?
> The pool chunks from ChunkCreator are deallocated while in pool because there
> is no reference to them
> -
>
> Key: HBASE-18375
> URL: https://issues.apache.org/jira/browse/HBASE-18375
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0-alpha-1
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
>Priority: Critical
> Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch,
> HBASE-18375-V03.patch, HBASE-18375-V04.patch
>
>
> Because MSLAB list of chunks was changed to list of chunk IDs, the chunks
> returned back to pool can be deallocated by JVM because there is no reference
> to them. The solution is to protect pool chunks from GC by the strong map of
> ChunkCreator introduced by HBASE-18010. Will prepare the patch today.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118229#comment-16118229 ] Anoop Sam John commented on HBASE-18375: This is a nice find. Now I have a doubt whether the performance numbers that we were seeing with MSLAB pool ON were right or not. The pool was not efficient with this bug. Will have to run them once again after this fix. > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16112173#comment-16112173 ] ramkrishna.s.vasudevan commented on HBASE-18375: [~st...@duboce.net] I have done my review. Am fine with the change (using strong ref always for chunks from pool) considering the fact what [~anastas] has found in her tests. How ever reading the code it seems to me that we have handled things but that seems not the case. I was just waiting for others reviews before we could commit this. > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111956#comment-16111956 ] stack commented on HBASE-18375: --- [~ram_krish] [~anoop.hbase] [~eshcar] What you all think? Patch looks reasonable. Nice digging [~anastas] > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16106401#comment-16106401 ] Anastasia Braginsky commented on HBASE-18375: - Hey [~eshcar], [~ram_krish], [~anoop.hbase], I want to bring your attention back to this JIRA. It includes three fixes that I believe need to be committed to the master. 1. Making pool chunks to be covered by strong map against GCing 2. Reversing the order of disconnecting the segment from the pipeline and this segment closing 3. Increasing the pipeline version upon flattening of the segment in it Please say your opinion regarding the commit. Thanks! > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094587#comment-16094587 ] Anastasia Braginsky commented on HBASE-18375: - Constant guarding the *only pool* chunks with strong Map is a clear solution, rather then relying on local thread references which may go wrong at least in the future code maintenance. Both reclaimedChunks and strongChunkIdMap has same data only when all chunks are returned to pool. Other than that reclaimedChunks purpose is to specify the chunks ready to be used, and strongChunkIdMap purpose is to protect from GC. However, I am OK to withdraw the change of protection of strong map. I am open to any suggestions that will come soon :) > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094533#comment-16094533 ] ramkrishna.s.vasudevan commented on HBASE-18375: If for sure CellChunkMap is saving from GC and only when we are putting it back to the pool we get this issue -from code perspective am not getting where things are going wrong (but may be Java's weakRef is something I don't get fully). Guarding the chunks with strong Map is definitely helping with NPEs. In case of pooled chunks and say initial Count and max Count is same - then we will have both reclaimedChunks and strongMapChunk with same data. I think if we are going to see this issue then going with the patch way is fine. Let see what others has to say. > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch, HBASE-18375-V04.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[
https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093172#comment-16093172
]
Hadoop QA commented on HBASE-18375:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
21s{color} | {color:red} hbase-server in master has 9 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
36m 3s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
1s{color} | {color:red} hbase-server generated 1 new + 9 unchanged - 0 fixed =
10 total (was 9) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}113m
36s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}167m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Increment of volatile field
org.apache.hadoop.hbase.regionserver.CompactionPipeline.version in
org.apache.hadoop.hbase.regionserver.CompactionPipeline.replaceAtIndex(int,
ImmutableSegment) At CompactionPipeline.java:in
org.apache.hadoop.hbase.regionserver.CompactionPipeline.replaceAtIndex(int,
ImmutableSegment) At CompactionPipeline.java:[line 284] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:757bf37 |
| JIRA Issue | HBASE-18375 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12877972/HBASE-18375-V04.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 8400b69b060d 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / f10f819 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC3 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/7716/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
|
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/7716/artifact/patchprocess/new-findbugs-hbase-server.html
|
| Test Results |
htt
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092819#comment-16092819 ] Anastasia Braginsky commented on HBASE-18375: - bq. Also the change in CompactionPipeline of removing and adding the last, now being done at the beginning is it intentional to avoid the NPE bug? The change in CompactionPipeline is in reversing the order of closing the suffix and disconnecting the suffix from the compaction pipeline. This problem wasn't seen, but generally it is correct to close the suffix AFTER it was already disconnected from pipeline. When the suffix is closed before disconnected from pipeline, the following problem can happen: 1. Segment is closed, assuming the reference count is zero, segment's chunks are released. 2. While the segment is still in the compaction pipeline, but after the chunks are released, scan appears and gets the reference to the segment. 3. The segment is disconnected from pipeline, but scan is accessing the released chunks... :( > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092737#comment-16092737 ] Anastasia Braginsky commented on HBASE-18375: - Hey Ram! Thanks for your questions, they make me rethink the process once again. I have not yet presented in this JIRA one more important problem introduced in HBASE-18010, where the flattening became a creation pattern. The pipeline version wasn't increased while flattening, so it allowed simultaneous change of the pipeline from the snapshot and from the flattening. I wanted to add this fix in other JIRA, but as I see it is all related, I will add this fix also here and I will publish a new patch soon. bq. So this chunk that we return from weak.get() at this point of time has a reference. If you are asking about the exact scenario happening in my failure, there the reference is removed from strongMap. But generally poolChunk can be also removed from weakMap. And yes the pool chunk still has the reference at this point in time ( weak.get() returns strong reference). bq. But when we are actually putting it to the reclaimedChunks, the chunk becomes null. In my scenario, I believe the pool chunk still has reference also there and it is added to reclaimedChunks in its "correct" form. But again if chunk is null then, and we don't add it back to reclaimedChunks, it is still not a desired behavior, because we deallocate chunks from pool and decreasing their number... bq. So when we do poll from reclaimedchunks we get a null ref and we just try to use that null ref and add it to the weakRefMap again and so we are working with null reference through out as per Again, I see the chunk is polled from reclaimedChunks when it still OK, but released later when it is already in weakRefMap. It is a subtle scenario, but causing a problem after all. To always keep pool chunks in strongMap is much cleaner and thus safer. bq. So in case of CellChunkMap atleast since we are sure to have a strong ref is it better to always return the strong ref in removeChunk() code rather than the weak ref (though it is available)? Sorry, I do not see why it is better to always return the strong ref in removeChunk() ... ? Currently removeChunk() returns strong reference anyway. > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch, > HBASE-18375-V03.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[
https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091367#comment-16091367
]
ramkrishna.s.vasudevan commented on HBASE-18375:
I think I get it now. Thanks for the explanation.
So here in removeChunk
{code}
Chunk removeChunk(int chunkId) {
WeakReference weak = this.weakChunkIdMap.remove(chunkId);
Chunk strong = this.strongChunkIdMap.remove(chunkId);
if (weak != null) {
return weak.get();
}
return strong;
}
{code}
So this chunk that we return from weak.get() at this point of time has a
reference. But when we are actually putting it to the reclaimedChunks ie.
{code}
Chunk chunk = ChunkCreator.this.removeChunk(chunkId);
if (chunk != null) {
if (chunk.isFromPool() && toAdd > 0) {
reclaimedChunks.add(chunk);
}
{code}
The chunk becomes null. So when we do poll from reclaimedchunks we get a null
ref and we just try to use that null ref and add it to the weakRefMap again and
so we are working with null reference through out as per
bq.And yes, it was still possible to poll the chunk from reclaimedChunks, but
it was deallocated while being in weakMap.
So in case of CellChunkMap atleast since we are sure to have a strong ref is it
better to always return the strong ref in removeChunk() code rather than the
weak ref (though it is available)?
Also the change in CompactionPipeline of removing and adding the last, now
being done at the beginning is it intentional to avoid the NPE bug?
> The pool chunks from ChunkCreator are deallocated while in pool because there
> is no reference to them
> -
>
> Key: HBASE-18375
> URL: https://issues.apache.org/jira/browse/HBASE-18375
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0-alpha-1
>Reporter: Anastasia Braginsky
>Priority: Critical
> Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch,
> HBASE-18375-V03.patch
>
>
> Because MSLAB list of chunks was changed to list of chunk IDs, the chunks
> returned back to pool can be deallocated by JVM because there is no reference
> to them. The solution is to protect pool chunks from GC by the strong map of
> ChunkCreator introduced by HBASE-18010. Will prepare the patch today.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[
https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091256#comment-16091256
]
Anastasia Braginsky commented on HBASE-18375:
-
Hey, [~ram_krish]! No problems, here are more explanations:
bq. So you say that the ByteBuffer inside the 'Chunk' becomes null and so
referencing the BB inside the chunk gives you NPE or is it the Chunk itself
being NULL? Can you paste the stack trace here?
What I see is the following scenario (I am running with CellChunkMap):
1. Chunk C is allocated from pool and is used as part of the Segment S. S is
currently part of the compaction pipeline. C is protected with strong pointer
as it is data chunk of the CellChunkMap.
2. Due to the snapshot of the pipeline, segment S is swapped out of the
pipeline.
3. S is closed, C is removed from strong map and is not referenced from
anywhere.
4. C is returned to the pool, but in parallel the GC is already freeing C.
5. As a result C's chunk ID is entered to weak map, but it references to null...
So I am getting null when I try to translate C's chunk ID. I will copy paste
here the stack, but it has heavy printouts all around. It was intensive
debugging till I understood that scenario.
{code}
2017-07-16 16:03:31,109 DEBUG
[RpcServer.default.FPBQ.Fifo.handler=24,queue=0,port=16020-inmemoryCompactions-1500216113902]
regionserver.CompactingMemStore: IN-MEMORY FLUSH: Pushing active segment into
compaction pipeline [Region:
usertable,user4599,1500212802830.bf1a03cc3ca0f1788720512a8e9275d0., Store:
values, values]
2017-07-16 16:03:31,109 DEBUG
[RpcServer.default.FPBQ.Fifo.handler=24,queue=0,port=16020-inmemoryCompactions-1500216113902]
regionserver.MemStoreCompactor: Starting the In-Memory Compaction for store
values
2017-07-16 16:03:31,109 DEBUG
[RpcServer.default.FPBQ.Fifo.handler=24,queue=0,port=16020-inmemoryCompactions-1500216113902]
regionserver.MemStoreCompactor: The youngest segment in the in-Memory
Compaction Pipeline for store values is going to be flattened to the CHUNK_MAP
2017-07-16 16:03:31,225 DEBUG
[RpcServer.default.FPBQ.Fifo.handler=24,queue=0,port=16020-inmemoryCompactions-1500216113902]
regionserver.CellChunkImmutableSegment: Number of new index chunks 3. The old
data chunks saved while flattening [1, 3, 15, 20, 32, 50, 51, 67, 72, 73, 75,
96, 121, 132, 155, 197, 199]
2017-07-16 16:03:31,236 DEBUG
[RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020] ipc.RpcServer:
callId: 820212 service: ClientService methodName: Get size: 128 connection:
10.73.119.93:44692 deadline: 9223372036854775807
java.io.IOException: java.lang.IllegalArgumentException:
<<< In CellChunkMap, cell must be associated with chunk with chunk ID 199 read
from offset 947764. We were looking for a cell at global index 47388 and index
inside chunk 47388, NUM_OF_CELL_REPS_IN_CHUNK: 104857.
<<< The index chunk in chunk array is Chunk@1992770427 allocs=0waste=2097148.
Number of chunks in the creator maps: 200. The chunk ID counter: 201. Verifying
once again week map: java.lang.ref.WeakReference@39f3b87f --> null.
<<< WEAK MAP: {198=java.lang.ref.WeakReference@44f41b54,
199=java.lang.ref.WeakReference@39f3b87f,
200=java.lang.ref.WeakReference@4a68f3b9}.
<<< STRONG MAP: {1=Chunk@116964628 allocs=13539waste=39, 2=Chunk@577555956
allocs=13539waste=92, 3=Chunk@1185303638 allocs=13539waste=36,
4=Chunk@1264486912 allocs=13547waste=151, 5=Chunk@143494754
allocs=13541waste=52, 6=Chunk@1279980010 allocs=13548waste=48,
7=Chunk@955957449 allocs=13539waste=50, 8=Chunk@122372956 allocs=13547waste=66,
9=Chunk@1666265929 allocs=13541waste=121, 10=Chunk@2022660968
allocs=13544waste=85, 11=Chunk@379396152 allocs=13539waste=57,
12=Chunk@1259454484 allocs=13544waste=90, 13=Chunk@556824900
allocs=13547waste=124, 14=Chunk@593895953 allocs=13547waste=107,
15=Chunk@1552239269 allocs=13539waste=58, 16=Chunk@424747073
allocs=13539waste=98, 17=Chunk@998867062 allocs=13540waste=126,
18=Chunk@695032764 allocs=13541waste=79, 19=Chunk@1598815318
allocs=13540waste=146, 20=Chunk@1334349014 allocs=13539waste=105,
21=Chunk@947203937 allocs=13547waste=82, 22=Chunk@2064003944
allocs=13544waste=102, 23=Chunk@2075121938 allocs=13541waste=20,
24=Chunk@1892287629 allocs=13544waste=119, 25=Chunk@1641362386
allocs=13539waste=146, 26=Chunk@721601011 allocs=13544waste=123,
27=Chunk@778107592 allocs=12waste=2095291, 28=Chunk@237923813
allocs=13541waste=26, 29=Chunk@1177388113 allocs=13547waste=108,
30=Chunk@2073942078 allocs=13539waste=130, 31=Chunk@1384303423
allocs=13544waste=68, 32=Chunk@801031631 allocs=13539waste=81,
33=Chunk@94094181 allocs=13545waste=16, 34=Chunk@282245056
allocs=13541waste=65, 35=Chunk@1414356438 allocs=13548waste=7,
36=Chunk@218397229 allocs=13540waste=17, 37=Chunk@1449066499
allocs=0waste=2097148, 38=Chunk@1404490175 allocs=13547waste=115,
39=Chunk@1831568882 allocs=13541waste=93,
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[
https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091170#comment-16091170
]
ramkrishna.s.vasudevan commented on HBASE-18375:
bq.3. When S is closed, C is returned to ChunkCreator, which in turn returns C
to the pool, but in parrallel the GC is already freeing C's "unreachable"
ByteBuffer.
and this
bq. However, while S is closed the chunks are released to ChunkCreator and the
following code is invoked:
So you say that the ByteBuffer inside the 'Chunk' becomes null and so
referencing the BB inside the chunk gives you NPE or is it the Chunk itself
being NULL? Can you paste the stack trace here?
{code}
Chunk chunk = ChunkCreator.this.removeChunk(chunkId);
if (chunk != null) {
if (chunk.isFromPool() && toAdd > 0) {
reclaimedChunks.add(chunk);
}
toAdd--;
}
{code}
So here if we have 'chunk' ref available then there is no problem and you see
that some time you don't get the chunk ref itself and so it is not getting
added to reclaimedChunks at all? And so next time you are not able to poll from
the reclaimedchunks?
If you are always going with strongChunkMap if there is a pool then that
'saveFromGC' also can be avoided while using CellChunkMap if there was a pool
already in place? Sorry for taking my time as I just want to be clear on the
problem. I may be missing something as you have already seen the issue but it
is just to be sure that we are on the same page.
> The pool chunks from ChunkCreator are deallocated while in pool because there
> is no reference to them
> -
>
> Key: HBASE-18375
> URL: https://issues.apache.org/jira/browse/HBASE-18375
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0-alpha-1
>Reporter: Anastasia Braginsky
>Priority: Critical
> Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch,
> HBASE-18375-V03.patch
>
>
> Because MSLAB list of chunks was changed to list of chunk IDs, the chunks
> returned back to pool can be deallocated by JVM because there is no reference
> to them. The solution is to protect pool chunks from GC by the strong map of
> ChunkCreator introduced by HBASE-18010. Will prepare the patch today.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[
https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090127#comment-16090127
]
Hadoop QA commented on HBASE-18375:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
50s{color} | {color:red} hbase-server in master has 9 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 0s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}123m
2s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}167m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 |
| JIRA Issue | HBASE-18375 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12877600/HBASE-18375-V03.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 9e0ba0617d06 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 2d5a0fb |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC3 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/7678/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/7678/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/7678/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> The pool chunks from ChunkCreator are deallocated while in pool because there
> is no reference to them
> -
>
> Key: HBASE-18375
> URL: https://issues.apache.or
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[
https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089794#comment-16089794
]
Hadoop QA commented on HBASE-18375:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
41s{color} | {color:red} hbase-server in master has 9 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 58s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 22m 2s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestMemStoreLAB |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:757bf37 |
| JIRA Issue | HBASE-18375 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12877565/HBASE-18375-V02.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 5fcfe918bcea 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 2d5a0fb |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC3 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/7675/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/7675/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/7675/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/7675/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This m
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089703#comment-16089703 ] Anastasia Braginsky commented on HBASE-18375: - Hey! I have answered the comments on the review board and uploaded new patch. Thanks! > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18375-V01.patch, HBASE-18375-V02.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[
https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089509#comment-16089509
]
Anastasia Braginsky commented on HBASE-18375:
-
bq. When there is a transfer to flat map to chunkMap we were converting weak to
strong ref right? Then why this problem happens? Sorry if am missing something.
Here is the detailed explanation of the problem, which is general and not
related to CellChunkMap directly. Let's assume the segments are implemented
with CellArrayMap and the following scenario happens:
1. Chunk C is allocated from pool and is used as part of the Segment S. S is
currently part of the compaction pipeline.
2. Due to the snapshot or the compaction of the pipeline, segment S is swapped
out of the pipeline. C is now unreachable by any reference. Because MSLAB is
holding only chunkIDs, Cells referencing to C are unreachable themselves, and
weak references from ChunkCreator map aren't considered by GC. Let us also
assume no scan was happening in parallel.
3. When S is closed, C is returned to ChunkCreator, which in turn returns C to
the pool, but in parrallel the GC is already freeing C's "unreachable"
ByteBuffer.
4. As a result the uninitialized chunk C is in the pool and is later allocated
to some other uses.
I personally hit this problem on the machine.
After the fix the problem didn't appear.
Regarding the CellChunkMap, you are right to say that there we shouldn't see
the problem, as CellChunkMap's data chunks are covered with strongMap and
shouldn't be released by GC. However, while S is closed the chunks are released
to ChunkCreator and the following code is invoked:
{code}
private void putbackChunks(Set chunks) {
int toAdd = Math.min(chunks.size(), this.maxCount -
reclaimedChunks.size());
Iterator iterator = chunks.iterator();
while (iterator.hasNext()) {
Integer chunkId = iterator.next();
// remove the chunks every time though they are from the pool or not
Chunk chunk = ChunkCreator.this.removeChunk(chunkId); // < the
chunk is disconnected either from weak or strong map, so there is a period of
time the chunk is uncoveread by any reference and still unreachable
if (chunk != null) {
if (chunk.isFromPool() && toAdd > 0) {
reclaimedChunks.add(chunk);
}
toAdd--;
}
}
}
{code}
Therefore the solution is to cover the pool chunks by strong map - forever.
Hope it is clearer now.
> The pool chunks from ChunkCreator are deallocated while in pool because there
> is no reference to them
> -
>
> Key: HBASE-18375
> URL: https://issues.apache.org/jira/browse/HBASE-18375
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0-alpha-1
>Reporter: Anastasia Braginsky
>Priority: Critical
> Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18375-V01.patch
>
>
> Because MSLAB list of chunks was changed to list of chunk IDs, the chunks
> returned back to pool can be deallocated by JVM because there is no reference
> to them. The solution is to protect pool chunks from GC by the strong map of
> ChunkCreator introduced by HBASE-18010. Will prepare the patch today.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089401#comment-16089401 ] ramkrishna.s.vasudevan commented on HBASE-18375: I just added some comments in RB. Putting the same here - When there is a transfer to flat map to chunkMap we were converting weak to strong ref right? Then why this problem happens? Sorry if am missing something. > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18375-V01.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16088831#comment-16088831 ] Anastasia Braginsky commented on HBASE-18375: - bq. +1 to fix there as well. How do you think it should be fixed there? Separate JIRA? Separate patch? In anyway, [~anoop.hbase], [~ram_krish], [~eshcar] please review the patch I have uploaded 2 days ago (here and on RB). So we can promote the fix at least in the master. Thanks!!! > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anastasia Braginsky >Priority: Critical > Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-18375-V01.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16086163#comment-16086163 ] Anoop Sam John commented on HBASE-18375: +1 to fix there as well. > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Priority: Critical > Attachments: HBASE-18375-V01.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085712#comment-16085712 ] Anastasia Braginsky commented on HBASE-18375: - I submit the patch with the fix, here and on RB. Please review. The solution is to always keep the pool chunks covered with strong map references protection. I was consistently hitting the null pointer exception caused by this problem. It happened when I was running with CellChunkMap. When running with this fix, the problem doesn't reappear. bq. That jira was committed long back (In April) and should be there in branch 2.0. Why? I was asking because I think that with HBASE-16438 and without the fix, the release 2.0 can be exposed to the problem with chunk being deallocated while transferred to pool. However, the chance of this to happen is very low because there the ChunkCreator map was holding soft references and not weak... So I wonder whether we need to do something there as well... [~anoop.hbase], [~ramkrishna], what do you think? > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Priority: Critical > Attachments: HBASE-18375-V01.patch > > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085599#comment-16085599 ] Anoop Sam John commented on HBASE-18375: That jira was committed long back (In April) and should be there in branch 2.0. Why? > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085581#comment-16085581 ] Anastasia Braginsky commented on HBASE-18375: - I think problem exists also with CellArrayMaps just the race condition is rarer. When we have a swap in compaction pipeline (due to compaction/merge/snapshot) the segment which is swapped out is no longer reachable (the cells themselves are no more reachable). Since then and till the chunks are returned back to pool by MSLAB, the chunks are exposed to GC activities. [~anoop.hbase], do you know whether HBASE-16438 was taken to the branch for release 2.0? > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085541#comment-16085541 ] Anoop Sam John commented on HBASE-18375: Those BB are referred by active Cells. Oh you mean when we make CellChunkMap where there are no more Cell pojos. Those cell data sit in some chunk BBs.. hmm !!! > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085524#comment-16085524 ] Anastasia Braginsky commented on HBASE-18375: - Because MSLAB holds list of chunkID, the ByteBuffer of the chunks (allocated by the pool) are collected by GC once there are no more references to the chunks. Those uninitialized chunks are later transferred to the pool and allocated from pool again... > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085519#comment-16085519 ] Anastasia Braginsky commented on HBASE-18375: - While the chunks are not in the pool, but in MSLAB they are deallocated > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18375) The pool chunks from ChunkCreator are deallocated while in pool because there is no reference to them
[ https://issues.apache.org/jira/browse/HBASE-18375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085518#comment-16085518 ] Anoop Sam John commented on HBASE-18375: We have Q of Chunk in MemStoreChunkPool where we keep the pooled chunks right? So what is the issue? private final BlockingQueue reclaimedChunks; > The pool chunks from ChunkCreator are deallocated while in pool because there > is no reference to them > - > > Key: HBASE-18375 > URL: https://issues.apache.org/jira/browse/HBASE-18375 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky > > Because MSLAB list of chunks was changed to list of chunk IDs, the chunks > returned back to pool can be deallocated by JVM because there is no reference > to them. The solution is to protect pool chunks from GC by the strong map of > ChunkCreator introduced by HBASE-18010. Will prepare the patch today. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
