[jira] [Commented] (HBASE-26142) NullPointerException when set 'hbase.hregion.memstore.mslab.indexchunksize.percent' to zero
[
https://issues.apache.org/jira/browse/HBASE-26142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17395450#comment-17395450
]
chenglei commented on HBASE-26142:
--
[~zhangduo],thank you very much for the review.
> NullPointerException when set
> 'hbase.hregion.memstore.mslab.indexchunksize.percent' to zero
> ---
>
> Key: HBASE-26142
> URL: https://issues.apache.org/jira/browse/HBASE-26142
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 2.4.0
>Reporter: chenglei
>Assignee: chenglei
>Priority: Critical
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.6, 2.3.7
>
>
> The default value of {{hbase.hregion.memstore.mslab.indexchunksize.percent}}
> introduced by HBASE-24892 is 0.1, but when we use {{DefaultMemStore}} by
> default , which has no {{IndexChunk}} and {{ChunkCreator.indexChunksPool}}
> is useless({{IndexChunk}} is only used by {{CompactingMemStore}}), so we set
> {{hbase.hregion.memstore.mslab.indexchunksize.percent}} to 0 to save memory
> space,
> But after running a while, the {{RegionServer}} throws
> {{NullPointerException}} and abort:
> {code:java}
>Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.ChunkCreator$MemStoreChunkPool.access$900(ChunkCreator.java:310)
> at
> org.apache.hadoop.hbase.regionserver.ChunkCreator.putbackChunks(ChunkCreator.java:608)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.recycleChunks(MemStoreLABImpl.java:297)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.close(MemStoreLABImpl.java:268)
> at
> org.apache.hadoop.hbase.regionserver.Segment.close(Segment.java:149)
> at
> org.apache.hadoop.hbase.regionserver.AbstractMemStore.clearSnapshot(AbstractMemStore.java:251)
> at
> org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1244)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$700(HStore.java:137)
> at
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2461)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2963)
> {code}
> The problem is caused by line 608 in {{ChunkCreator.putbackChunks}} :
> {{Chunk.isIndexChunk}} incorrectly returns true for {{DataChunk}} and
> unexpectedly invokes {{indexChunksPool.putbackChunks}}， while
> {{indexChunksPool}} is null:
> {code:java}
> 594 synchronized void putbackChunks(Set chunks) {
> 595// if there is no pool just try to clear the chunkIdMap in case there
> is something
> 596if (dataChunksPool == null && indexChunksPool == null) {
> 597 this.removeChunks(chunks);
> 598 return;
> 599}
> 600
> 601 // if there is a pool, go over all chunk IDs that came back, the chunks
> may be from pool or not
> 602for (int chunkID : chunks) {
> 603 // translate chunk ID to chunk, if chunk initially wasn't in pool
> 604 // this translation will (most likely) return null
> 605 Chunk chunk = ChunkCreator.this.getChunk(chunkID);
> 606 if (chunk != null) {
> 607if (chunk.isFromPool() && chunk.isIndexChunk()) {
> 608 indexChunksPool.putbackChunks(chunk);
> {code}
> For {{DataChunk}} , {{Chunk.isIndexChunk}} return true because
> {{Chunk.isIndexChunk}} determines the type of {{chunk}} based on
> {{Chunk.size}}
> {code:java}
> boolean isIndexChunk() {
> return size ==
> ChunkCreator.getInstance().getChunkSize(ChunkCreator.ChunkType.INDEX_CHUNK);
> }
> {code}
> and {{ChunkCreator.getChunkSize}} incorrectly return {{DataChunk}} size when
> {{ChunkCreator.indexChunksPool}} is null:
> {code:java}
> int getChunkSize(ChunkType chunkType) {
> switch (chunkType) {
> case INDEX_CHUNK:
> if (indexChunksPool != null) {
> return indexChunksPool.getChunkSize();
> }
> case DATA_CHUNK:
> if (dataChunksPool != null) {
> return dataChunksPool.getChunkSize();
> } else { // When pools are empty
> return chunkSize;
> }
> default:
> throw new IllegalArgumentException(
> "chunkType must either be INDEX_CHUNK or DATA_CHUNK");
> }
> }
> {code}
> In my opinion, in addition to erroneous implementation of
> {{ChunkCreator.getChunkSize}}, we would better not determine the type of
> {{Chunk}} based on {{Chunk.size}}, because
> {{hbase.hregion.memstore.mslab.indexchunksize.percent}} is set by user and
> the size of {{IndexChunk}} and {{DataChunk}} could be the same.Tagged a
> {{ChunkType}} to {{Chunk}} is a better choice.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-26142) NullPointerException when set 'hbase.hregion.memstore.mslab.indexchunksize.percent' to zero
[
https://issues.apache.org/jira/browse/HBASE-26142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394939#comment-17394939
]
Hudson commented on HBASE-26142:
Results for branch branch-2
[build #315 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/315/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/315/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/315/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/315/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/315/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/315//console].
> NullPointerException when set
> 'hbase.hregion.memstore.mslab.indexchunksize.percent' to zero
> ---
>
> Key: HBASE-26142
> URL: https://issues.apache.org/jira/browse/HBASE-26142
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 2.4.0
>Reporter: chenglei
>Assignee: chenglei
>Priority: Critical
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.6, 2.3.7
>
>
> The default value of {{hbase.hregion.memstore.mslab.indexchunksize.percent}}
> introduced by HBASE-24892 is 0.1, but when we use {{DefaultMemStore}} by
> default , which has no {{IndexChunk}} and {{ChunkCreator.indexChunksPool}}
> is useless({{IndexChunk}} is only used by {{CompactingMemStore}}), so we set
> {{hbase.hregion.memstore.mslab.indexchunksize.percent}} to 0 to save memory
> space,
> But after running a while, the {{RegionServer}} throws
> {{NullPointerException}} and abort:
> {code:java}
>Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.ChunkCreator$MemStoreChunkPool.access$900(ChunkCreator.java:310)
> at
> org.apache.hadoop.hbase.regionserver.ChunkCreator.putbackChunks(ChunkCreator.java:608)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.recycleChunks(MemStoreLABImpl.java:297)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.close(MemStoreLABImpl.java:268)
> at
> org.apache.hadoop.hbase.regionserver.Segment.close(Segment.java:149)
> at
> org.apache.hadoop.hbase.regionserver.AbstractMemStore.clearSnapshot(AbstractMemStore.java:251)
> at
> org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1244)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$700(HStore.java:137)
> at
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2461)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2963)
> {code}
> The problem is caused by line 608 in {{ChunkCreator.putbackChunks}} :
> {{Chunk.isIndexChunk}} incorrectly returns true for {{DataChunk}} and
> unexpectedly invokes {{indexChunksPool.putbackChunks}}， while
> {{indexChunksPool}} is null:
> {code:java}
> 594 synchronized void putbackChunks(Set chunks) {
> 595// if there is no pool just try to clear the chunkIdMap in case there
> is something
> 596if (dataChunksPool == null && indexChunksPool == null) {
> 597 this.removeChunks(chunks);
> 598 return;
> 599}
> 600
> 601 // if there is a pool, go over all chunk IDs that came back, the chunks
> may be from pool or not
> 602for (int chunkID : chunks) {
> 603 // translate chunk ID to chunk, if chunk initially wasn't in pool
> 604 // this translation will (most likely) return null
> 605 Chunk chunk = ChunkCreator.this.getChunk(chunkID);
> 606 if (chunk != null) {
> 607if (chunk.isFromPool() && chunk.isIndexChunk()) {
> 608 indexChunksPool.putbackChunks(chunk);
> {code}
> For {{DataChunk}} , {{Chunk.isIndexChunk}} return true because
> {{Chunk.isIndexChunk}} determines the type of {{chunk}} based on
> {{Chunk.size}}
> {code:java}
> boolean
[jira] [Commented] (HBASE-26142) NullPointerException when set 'hbase.hregion.memstore.mslab.indexchunksize.percent' to zero
[
https://issues.apache.org/jira/browse/HBASE-26142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394887#comment-17394887
]
Hudson commented on HBASE-26142:
Results for branch branch-2.3
[build #269 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/269/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/269/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/269/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/269/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/269/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> NullPointerException when set
> 'hbase.hregion.memstore.mslab.indexchunksize.percent' to zero
> ---
>
> Key: HBASE-26142
> URL: https://issues.apache.org/jira/browse/HBASE-26142
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 2.4.0
>Reporter: chenglei
>Assignee: chenglei
>Priority: Critical
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.6, 2.3.7
>
>
> The default value of {{hbase.hregion.memstore.mslab.indexchunksize.percent}}
> introduced by HBASE-24892 is 0.1, but when we use {{DefaultMemStore}} by
> default , which has no {{IndexChunk}} and {{ChunkCreator.indexChunksPool}}
> is useless({{IndexChunk}} is only used by {{CompactingMemStore}}), so we set
> {{hbase.hregion.memstore.mslab.indexchunksize.percent}} to 0 to save memory
> space,
> But after running a while, the {{RegionServer}} throws
> {{NullPointerException}} and abort:
> {code:java}
>Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.ChunkCreator$MemStoreChunkPool.access$900(ChunkCreator.java:310)
> at
> org.apache.hadoop.hbase.regionserver.ChunkCreator.putbackChunks(ChunkCreator.java:608)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.recycleChunks(MemStoreLABImpl.java:297)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.close(MemStoreLABImpl.java:268)
> at
> org.apache.hadoop.hbase.regionserver.Segment.close(Segment.java:149)
> at
> org.apache.hadoop.hbase.regionserver.AbstractMemStore.clearSnapshot(AbstractMemStore.java:251)
> at
> org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1244)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$700(HStore.java:137)
> at
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2461)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2963)
> {code}
> The problem is caused by line 608 in {{ChunkCreator.putbackChunks}} :
> {{Chunk.isIndexChunk}} incorrectly returns true for {{DataChunk}} and
> unexpectedly invokes {{indexChunksPool.putbackChunks}}， while
> {{indexChunksPool}} is null:
> {code:java}
> 594 synchronized void putbackChunks(Set chunks) {
> 595// if there is no pool just try to clear the chunkIdMap in case there
> is something
> 596if (dataChunksPool == null && indexChunksPool == null) {
> 597 this.removeChunks(chunks);
> 598 return;
> 599}
> 600
> 601 // if there is a pool, go over all chunk IDs that came back, the chunks
> may be from pool or not
> 602for (int chunkID : chunks) {
> 603 // translate chunk ID to chunk, if chunk initially wasn't in pool
> 604 // this translation will (most likely) return null
> 605 Chunk chunk = ChunkCreator.this.getChunk(chunkID);
> 606 if (chunk != null) {
> 607if (chunk.isFromPool() && chunk.isIndexChunk()) {
> 608 indexChunksPool.putbackChunks(chunk);
> {code}
> For {{DataChunk}} , {{Chunk.isIndexChunk}} return true because
> {{Chunk.isIndexChunk}} determines the type of {{chunk}} based on
> {{Chunk.size}}
> {code:java}
> boolean isIndexChunk() {
> return size ==
> ChunkCreator.getInstance().getChunkSize(ChunkCreator.ChunkType.INDEX_CHUNK);
> }
> {code}
> and
[jira] [Commented] (HBASE-26142) NullPointerException when set 'hbase.hregion.memstore.mslab.indexchunksize.percent' to zero
[
https://issues.apache.org/jira/browse/HBASE-26142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394744#comment-17394744
]
Hudson commented on HBASE-26142:
Results for branch master
[build #361 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/361/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/361/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/361/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/361/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> NullPointerException when set
> 'hbase.hregion.memstore.mslab.indexchunksize.percent' to zero
> ---
>
> Key: HBASE-26142
> URL: https://issues.apache.org/jira/browse/HBASE-26142
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 2.4.0
>Reporter: chenglei
>Assignee: chenglei
>Priority: Critical
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.6, 2.3.7
>
>
> The default value of {{hbase.hregion.memstore.mslab.indexchunksize.percent}}
> introduced by HBASE-24892 is 0.1, but when we use {{DefaultMemStore}} by
> default , which has no {{IndexChunk}} and {{ChunkCreator.indexChunksPool}}
> is useless({{IndexChunk}} is only used by {{CompactingMemStore}}), so we set
> {{hbase.hregion.memstore.mslab.indexchunksize.percent}} to 0 to save memory
> space,
> But after running a while, the {{RegionServer}} throws
> {{NullPointerException}} and abort:
> {code:java}
>Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.ChunkCreator$MemStoreChunkPool.access$900(ChunkCreator.java:310)
> at
> org.apache.hadoop.hbase.regionserver.ChunkCreator.putbackChunks(ChunkCreator.java:608)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.recycleChunks(MemStoreLABImpl.java:297)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.close(MemStoreLABImpl.java:268)
> at
> org.apache.hadoop.hbase.regionserver.Segment.close(Segment.java:149)
> at
> org.apache.hadoop.hbase.regionserver.AbstractMemStore.clearSnapshot(AbstractMemStore.java:251)
> at
> org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1244)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$700(HStore.java:137)
> at
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2461)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2963)
> {code}
> The problem is caused by line 608 in {{ChunkCreator.putbackChunks}} :
> {{Chunk.isIndexChunk}} incorrectly returns true for {{DataChunk}} and
> unexpectedly invokes {{indexChunksPool.putbackChunks}}， while
> {{indexChunksPool}} is null:
> {code:java}
> 594 synchronized void putbackChunks(Set chunks) {
> 595// if there is no pool just try to clear the chunkIdMap in case there
> is something
> 596if (dataChunksPool == null && indexChunksPool == null) {
> 597 this.removeChunks(chunks);
> 598 return;
> 599}
> 600
> 601 // if there is a pool, go over all chunk IDs that came back, the chunks
> may be from pool or not
> 602for (int chunkID : chunks) {
> 603 // translate chunk ID to chunk, if chunk initially wasn't in pool
> 604 // this translation will (most likely) return null
> 605 Chunk chunk = ChunkCreator.this.getChunk(chunkID);
> 606 if (chunk != null) {
> 607if (chunk.isFromPool() && chunk.isIndexChunk()) {
> 608 indexChunksPool.putbackChunks(chunk);
> {code}
> For {{DataChunk}} , {{Chunk.isIndexChunk}} return true because
> {{Chunk.isIndexChunk}} determines the type of {{chunk}} based on
> {{Chunk.size}}
> {code:java}
> boolean isIndexChunk() {
> return size ==
> ChunkCreator.getInstance().getChunkSize(ChunkCreator.ChunkType.INDEX_CHUNK);
> }
> {code}
> and {{ChunkCreator.getChunkSize}} incorrectly return {{DataChunk}} size when
> {{ChunkCreator.indexChunksPool}} is null:
> {code:java}
> int getChunkSize(ChunkType chunkType) {
> switch (chunkType) {
> case INDEX_CHUNK:
> if (indexChunksPool != null) {
[jira] [Commented] (HBASE-26142) NullPointerException when set 'hbase.hregion.memstore.mslab.indexchunksize.percent' to zero
[
https://issues.apache.org/jira/browse/HBASE-26142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394422#comment-17394422
]
Hudson commented on HBASE-26142:
Results for branch branch-2.4
[build #175 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/175/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/175/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/175/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/175/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/175/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> NullPointerException when set
> 'hbase.hregion.memstore.mslab.indexchunksize.percent' to zero
> ---
>
> Key: HBASE-26142
> URL: https://issues.apache.org/jira/browse/HBASE-26142
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 2.4.0
>Reporter: chenglei
>Assignee: chenglei
>Priority: Critical
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.6, 2.3.7
>
>
> The default value of {{hbase.hregion.memstore.mslab.indexchunksize.percent}}
> introduced by HBASE-24892 is 0.1, but when we use {{DefaultMemStore}} by
> default , which has no {{IndexChunk}} and {{ChunkCreator.indexChunksPool}}
> is useless({{IndexChunk}} is only used by {{CompactingMemStore}}), so we set
> {{hbase.hregion.memstore.mslab.indexchunksize.percent}} to 0 to save memory
> space,
> But after running a while, the {{RegionServer}} throws
> {{NullPointerException}} and abort:
> {code:java}
>Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.ChunkCreator$MemStoreChunkPool.access$900(ChunkCreator.java:310)
> at
> org.apache.hadoop.hbase.regionserver.ChunkCreator.putbackChunks(ChunkCreator.java:608)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.recycleChunks(MemStoreLABImpl.java:297)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.close(MemStoreLABImpl.java:268)
> at
> org.apache.hadoop.hbase.regionserver.Segment.close(Segment.java:149)
> at
> org.apache.hadoop.hbase.regionserver.AbstractMemStore.clearSnapshot(AbstractMemStore.java:251)
> at
> org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1244)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$700(HStore.java:137)
> at
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2461)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2963)
> {code}
> The problem is caused by line 608 in {{ChunkCreator.putbackChunks}} :
> {{Chunk.isIndexChunk}} incorrectly returns true for {{DataChunk}} and
> unexpectedly invokes {{indexChunksPool.putbackChunks}}， while
> {{indexChunksPool}} is null:
> {code:java}
> 594 synchronized void putbackChunks(Set chunks) {
> 595// if there is no pool just try to clear the chunkIdMap in case there
> is something
> 596if (dataChunksPool == null && indexChunksPool == null) {
> 597 this.removeChunks(chunks);
> 598 return;
> 599}
> 600
> 601 // if there is a pool, go over all chunk IDs that came back, the chunks
> may be from pool or not
> 602for (int chunkID : chunks) {
> 603 // translate chunk ID to chunk, if chunk initially wasn't in pool
> 604 // this translation will (most likely) return null
> 605 Chunk chunk = ChunkCreator.this.getChunk(chunkID);
> 606 if (chunk != null) {
> 607if (chunk.isFromPool() && chunk.isIndexChunk()) {
> 608 indexChunksPool.putbackChunks(chunk);
> {code}
> For {{DataChunk}} , {{Chunk.isIndexChunk}} return true because
> {{Chunk.isIndexChunk}} determines the type of {{chunk}} based on
> {{Chunk.size}}
> {code:java}
> boolean isIndexChunk() {
> return size ==
> ChunkCreator.getInstance().getChunkSize(ChunkCreator.ChunkType.INDEX_CHUNK);
> }
> {code}
> and
[jira] [Commented] (HBASE-26142) NullPointerException when set 'hbase.hregion.memstore.mslab.indexchunksize.percent' to zero
[
https://issues.apache.org/jira/browse/HBASE-26142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387925#comment-17387925
]
chenglei commented on HBASE-26142:
--
[~anoop.hbase], the default value of
{{hbase.hregion.memstore.mslab.indexchunksize.percent}} is 0.1, that is to say
, the byte size of the index chunk pool is
{hbase.regionserver.global.memstore.size}*0.1 by default
> NullPointerException when set
> 'hbase.hregion.memstore.mslab.indexchunksize.percent' to zero
> ---
>
> Key: HBASE-26142
> URL: https://issues.apache.org/jira/browse/HBASE-26142
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 2.4.0
>Reporter: chenglei
>Assignee: chenglei
>Priority: Critical
>
> By default, we use {{DefaultMemStore}} , which use no {{IndexChunk}} and
> {{ChunkCreator.indexChunksPool}} is useless, so we could set
> {{hbase.hregion.memstore.mslab.indexchunksize.percent}} to 0 to save memory
> space,
> But after running a while, the {{RegionServer}} throws
> {{NullPointerException}} and abort:
> {code:java}
>Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.ChunkCreator$MemStoreChunkPool.access$900(ChunkCreator.java:310)
> at
> org.apache.hadoop.hbase.regionserver.ChunkCreator.putbackChunks(ChunkCreator.java:608)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.recycleChunks(MemStoreLABImpl.java:297)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.close(MemStoreLABImpl.java:268)
> at
> org.apache.hadoop.hbase.regionserver.Segment.close(Segment.java:149)
> at
> org.apache.hadoop.hbase.regionserver.AbstractMemStore.clearSnapshot(AbstractMemStore.java:251)
> at
> org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1244)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$700(HStore.java:137)
> at
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2461)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2963)
> {code}
> The problem is caused by line 608 in {{ChunkCreator.putbackChunks}}
> :{{Chunk.isIndexChunk}} return true for{{DataChunk}} , and
> {{ChunkCreator.indexChunksPool}} is null:
> {code:java}
> 594 synchronized void putbackChunks(Set chunks) {
> 595// if there is no pool just try to clear the chunkIdMap in case there
> is something
> 596if (dataChunksPool == null && indexChunksPool == null) {
> 597 this.removeChunks(chunks);
> 598 return;
> 599}
> 600
> 601 // if there is a pool, go over all chunk IDs that came back, the chunks
> may be from pool or not
> 602for (int chunkID : chunks) {
> 603 // translate chunk ID to chunk, if chunk initially wasn't in pool
> 604 // this translation will (most likely) return null
> 605 Chunk chunk = ChunkCreator.this.getChunk(chunkID);
> 606 if (chunk != null) {
> 607if (chunk.isFromPool() && chunk.isIndexChunk()) {
> 608 indexChunksPool.putbackChunks(chunk);
> {code}
> For {{DataChunk}} , {{Chunk.isIndexChunk}} return true because
> {{Chunk.isIndexChunk}} determines the type of {{chunk}} based on
> {{Chunk.size}}
> {code:java}
> boolean isIndexChunk() {
> return size ==
> ChunkCreator.getInstance().getChunkSize(ChunkCreator.ChunkType.INDEX_CHUNK);
> }
> {code}
> and {{ChunkCreator.getChunkSize}} incorrectly return {{DataChunk}} size when
> {{ChunkCreator.indexChunksPool}} is null:
> {code:java}
> int getChunkSize(ChunkType chunkType) {
> switch (chunkType) {
> case INDEX_CHUNK:
> if (indexChunksPool != null) {
> return indexChunksPool.getChunkSize();
> }
> case DATA_CHUNK:
> if (dataChunksPool != null) {
> return dataChunksPool.getChunkSize();
> } else { // When pools are empty
> return chunkSize;
> }
> default:
> throw new IllegalArgumentException(
> "chunkType must either be INDEX_CHUNK or DATA_CHUNK");
> }
> }
> {code}
> In my opinion, in addition to erroneous implementation of
> {{ChunkCreator.getChunkSize}}, we would better not determine the type of
> {{Chunk}} based on {{Chunk.size}}, because
> {{hbase.hregion.memstore.mslab.indexchunksize.percent}} is set by user and
> the size of {{IndexChunk}} and {{DataChunk}} could be the same.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-26142) NullPointerException when set 'hbase.hregion.memstore.mslab.indexchunksize.percent' to zero
[
https://issues.apache.org/jira/browse/HBASE-26142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17387905#comment-17387905
]
Anoop Sam John commented on HBASE-26142:
what is the default BB pool size that this index chunk pool will create?
> NullPointerException when set
> 'hbase.hregion.memstore.mslab.indexchunksize.percent' to zero
> ---
>
> Key: HBASE-26142
> URL: https://issues.apache.org/jira/browse/HBASE-26142
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 2.4.0
>Reporter: chenglei
>Assignee: chenglei
>Priority: Critical
>
> By default, we use {{DefaultMemStore}} , which use no {{IndexChunk}} and
> {{ChunkCreator.indexChunksPool}} is useless, so we could set
> {{hbase.hregion.memstore.mslab.indexchunksize.percent}} to 0 to save memory
> space,
> But after running a while, the {{RegionServer}} throws
> {{NullPointerException}} and abort:
> {code:java}
>Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.ChunkCreator$MemStoreChunkPool.access$900(ChunkCreator.java:310)
> at
> org.apache.hadoop.hbase.regionserver.ChunkCreator.putbackChunks(ChunkCreator.java:608)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.recycleChunks(MemStoreLABImpl.java:297)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.close(MemStoreLABImpl.java:268)
> at
> org.apache.hadoop.hbase.regionserver.Segment.close(Segment.java:149)
> at
> org.apache.hadoop.hbase.regionserver.AbstractMemStore.clearSnapshot(AbstractMemStore.java:251)
> at
> org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1244)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$700(HStore.java:137)
> at
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2461)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2963)
> {code}
> The problem is caused by line 608 in {{ChunkCreator.putbackChunks}}
> :{{Chunk.isIndexChunk}} return true for{{DataChunk}} , and
> {{ChunkCreator.indexChunksPool}} is null:
> {code:java}
> 594 synchronized void putbackChunks(Set chunks) {
> 595// if there is no pool just try to clear the chunkIdMap in case there
> is something
> 596if (dataChunksPool == null && indexChunksPool == null) {
> 597 this.removeChunks(chunks);
> 598 return;
> 599}
> 600
> 601 // if there is a pool, go over all chunk IDs that came back, the chunks
> may be from pool or not
> 602for (int chunkID : chunks) {
> 603 // translate chunk ID to chunk, if chunk initially wasn't in pool
> 604 // this translation will (most likely) return null
> 605 Chunk chunk = ChunkCreator.this.getChunk(chunkID);
> 606 if (chunk != null) {
> 607if (chunk.isFromPool() && chunk.isIndexChunk()) {
> 608 indexChunksPool.putbackChunks(chunk);
> {code}
> For {{DataChunk}} , {{Chunk.isIndexChunk}} return true because
> {{Chunk.isIndexChunk}} determines the type of {{chunk}} based on
> {{Chunk.size}}
> {code:java}
> boolean isIndexChunk() {
> return size ==
> ChunkCreator.getInstance().getChunkSize(ChunkCreator.ChunkType.INDEX_CHUNK);
> }
> {code}
> and {{ChunkCreator.getChunkSize}} incorrectly return {{DataChunk}} size when
> {{ChunkCreator.indexChunksPool}} is null:
> {code:java}
> int getChunkSize(ChunkType chunkType) {
> switch (chunkType) {
> case INDEX_CHUNK:
> if (indexChunksPool != null) {
> return indexChunksPool.getChunkSize();
> }
> case DATA_CHUNK:
> if (dataChunksPool != null) {
> return dataChunksPool.getChunkSize();
> } else { // When pools are empty
> return chunkSize;
> }
> default:
> throw new IllegalArgumentException(
> "chunkType must either be INDEX_CHUNK or DATA_CHUNK");
> }
> }
> {code}
> In my opinion, in addition to erroneous implementation of
> {{ChunkCreator.getChunkSize}}, we would better not determine the type of
> {{Chunk}} based on {{Chunk.size}}, because
> {{hbase.hregion.memstore.mslab.indexchunksize.percent}} is set by user and
> the size of {{IndexChunk}} and {{DataChunk}} could be the same.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
