[ https://issues.apache.org/jira/browse/HBASE-26142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17394422#comment-17394422 ]
Hudson commented on HBASE-26142: -------------------------------- Results for branch branch-2.4 [build #175 on builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/175/]: (x) *{color:red}-1 overall{color}* ---- details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/175/General_20Nightly_20Build_20Report/] (x) {color:red}-1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/175/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/] (x) {color:red}-1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/175/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (x) {color:red}-1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/175/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > NullPointerException when set > 'hbase.hregion.memstore.mslab.indexchunksize.percent' to zero > ------------------------------------------------------------------------------------------- > > Key: HBASE-26142 > URL: https://issues.apache.org/jira/browse/HBASE-26142 > Project: HBase > Issue Type: Bug > Affects Versions: 3.0.0-alpha-1, 2.4.0 > Reporter: chenglei > Assignee: chenglei > Priority: Critical > Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.6, 2.3.7 > > > The default value of {{hbase.hregion.memstore.mslab.indexchunksize.percent}} > introduced by HBASE-24892 is 0.1, but when we use {{DefaultMemStore}} by > default , which has no {{IndexChunk}} and {{ChunkCreator.indexChunksPool}} > is useless({{IndexChunk}} is only used by {{CompactingMemStore}}), so we set > {{hbase.hregion.memstore.mslab.indexchunksize.percent}} to 0 to save memory > space, > But after running a while, the {{RegionServer}} throws > {{NullPointerException}} and abort: > {code:java} > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hbase.regionserver.ChunkCreator$MemStoreChunkPool.access$900(ChunkCreator.java:310) > at > org.apache.hadoop.hbase.regionserver.ChunkCreator.putbackChunks(ChunkCreator.java:608) > at > org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.recycleChunks(MemStoreLABImpl.java:297) > at > org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.close(MemStoreLABImpl.java:268) > at > org.apache.hadoop.hbase.regionserver.Segment.close(Segment.java:149) > at > org.apache.hadoop.hbase.regionserver.AbstractMemStore.clearSnapshot(AbstractMemStore.java:251) > at > org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1244) > at > org.apache.hadoop.hbase.regionserver.HStore.access$700(HStore.java:137) > at > org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2461) > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2963) > {code} > The problem is caused by line 608 in {{ChunkCreator.putbackChunks}} : > {{Chunk.isIndexChunk}} incorrectly returns true for {{DataChunk}} and > unexpectedly invokes {{indexChunksPool.putbackChunks}}, while > {{indexChunksPool}} is null: > {code:java} > 594 synchronized void putbackChunks(Set<Integer> chunks) { > 595 // if there is no pool just try to clear the chunkIdMap in case there > is something > 596 if (dataChunksPool == null && indexChunksPool == null) { > 597 this.removeChunks(chunks); > 598 return; > 599 } > 600 > 601 // if there is a pool, go over all chunk IDs that came back, the chunks > may be from pool or not > 602 for (int chunkID : chunks) { > 603 // translate chunk ID to chunk, if chunk initially wasn't in pool > 604 // this translation will (most likely) return null > 605 Chunk chunk = ChunkCreator.this.getChunk(chunkID); > 606 if (chunk != null) { > 607 if (chunk.isFromPool() && chunk.isIndexChunk()) { > 608 indexChunksPool.putbackChunks(chunk); > {code} > For {{DataChunk}} , {{Chunk.isIndexChunk}} return true because > {{Chunk.isIndexChunk}} determines the type of {{chunk}} based on > {{Chunk.size}} > {code:java} > boolean isIndexChunk() { > return size == > ChunkCreator.getInstance().getChunkSize(ChunkCreator.ChunkType.INDEX_CHUNK); > } > {code} > and {{ChunkCreator.getChunkSize}} incorrectly return {{DataChunk}} size when > {{ChunkCreator.indexChunksPool}} is null: > {code:java} > int getChunkSize(ChunkType chunkType) { > switch (chunkType) { > case INDEX_CHUNK: > if (indexChunksPool != null) { > return indexChunksPool.getChunkSize(); > } > case DATA_CHUNK: > if (dataChunksPool != null) { > return dataChunksPool.getChunkSize(); > } else { // When pools are empty > return chunkSize; > } > default: > throw new IllegalArgumentException( > "chunkType must either be INDEX_CHUNK or DATA_CHUNK"); > } > } > {code} > In my opinion, in addition to erroneous implementation of > {{ChunkCreator.getChunkSize}}, we would better not determine the type of > {{Chunk}} based on {{Chunk.size}}, because > {{hbase.hregion.memstore.mslab.indexchunksize.percent}} is set by user and > the size of {{IndexChunk}} and {{DataChunk}} could be the same.Tagged a > {{ChunkType}} to {{Chunk}} is a better choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)