[jira] [Commented] (HBASE-15243) Utilize the lowest seek value when all Filters in MUST_PASS_ONE FilterList return SEEK_NEXT_USING_HINT
[ https://issues.apache.org/jira/browse/HBASE-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184595#comment-15184595 ] Hudson commented on HBASE-15243: SUCCESS: Integrated in HBase-1.3-IT #543 (See [https://builds.apache.org/job/HBase-1.3-IT/543/]) HBASE-15243 Utilize the lowest seek value when all Filters in (tedyu: rev 4ec6ad12c9199ab39b5650aa62cece31672f838f) * hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilterListOrOperatorWithBlkCnt.java * hbase-client/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java > Utilize the lowest seek value when all Filters in MUST_PASS_ONE FilterList > return SEEK_NEXT_USING_HINT > -- > > Key: HBASE-15243 > URL: https://issues.apache.org/jira/browse/HBASE-15243 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15243-v1.txt, HBASE-15243-v2.txt, > HBASE-15243-v3.txt, HBASE-15243-v4.txt, HBASE-15243-v5.txt, > HBASE-15243-v6.txt, HBASE-15243-v7.txt > > > As Preston Koprivica pointed out at the tail of HBASE-4394, when all filters > in a MUST_PASS_ONE FilterList return a SEEK_USING_NEXT_HINT code, we should > return SEEK_NEXT_USING_HINT from the FilterList to utilize the lowest seek > value. > This would save unnecessary accesses to blocks which are not in scan result. > A unit test has been added which shows the reduction in the number of blocks > accessed when this optimization takes effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15413) Procedure-V2: print out ProcedureInfo during trace
[ https://issues.apache.org/jira/browse/HBASE-15413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184594#comment-15184594 ] Hudson commented on HBASE-15413: SUCCESS: Integrated in HBase-1.3-IT #543 (See [https://builds.apache.org/job/HBase-1.3-IT/543/]) HBASE-15413 Procedure-V2: print out ProcedureInfo during trace (Stephen (syuanjiangdev: rev 0a025a1aca14abc5deaeefc61d977bdb1fc49161) * hbase-common/src/main/java/org/apache/hadoop/hbase/ProcedureInfo.java > Procedure-V2: print out ProcedureInfo during trace > -- > > Key: HBASE-15413 > URL: https://issues.apache.org/jira/browse/HBASE-15413 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Affects Versions: 1.2.0, 1.1.4 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang >Priority: Trivial > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.4.0 > > Attachments: HBASE-15413.v1-master.patch > > > Before the HBASE-15100 refactored the code to print the ProcedureInfo object, > we don't have the needs. Now we need to implement a customized toString() > function to print out information in the ProcedureInfo object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15261) Make Throwable t in DaughterOpener volatile
[ https://issues.apache.org/jira/browse/HBASE-15261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184593#comment-15184593 ] Hudson commented on HBASE-15261: SUCCESS: Integrated in HBase-1.3 #591 (See [https://builds.apache.org/job/HBase-1.3/591/]) HBASE-15261 Make Throwable t in DaughterOpener volatile (Huaxiang Sun) (stack: rev 3b970eeb09f040caf91ef780c28c6c5f50e4fe57) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransactionImpl.java > Make Throwable t in DaughterOpener volatile > --- > > Key: HBASE-15261 > URL: https://issues.apache.org/jira/browse/HBASE-15261 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.4.0 > > Attachments: HBASE-15261-001.patch, HBASE-15261-001.patch > > > In the region split process, daughter regions are opened in different > threads, Throwable t is set in these threads and it is checked in the calling > thread. Need to make it volatile so the checking will not miss any exceptions > from opening daughter regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15354) Use same criteria for clearing meta cache for all operations
[ https://issues.apache.org/jira/browse/HBASE-15354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184592#comment-15184592 ] Hudson commented on HBASE-15354: SUCCESS: Integrated in HBase-1.3 #591 (See [https://builds.apache.org/job/HBase-1.3/591/]) HBASE-15354 Use same criteria for clearing meta cache for all operations (antonov: rev 944c5bf67d8822474cde36e1157c658d9ea2f674) * hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionServerCallable.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionAdminServiceCallable.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMetaCache.java > Use same criteria for clearing meta cache for all operations > > > Key: HBASE-15354 > URL: https://issues.apache.org/jira/browse/HBASE-15354 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 1.2.1 > > Attachments: HBASE-15354-V0.patch, HBASE-15354-V1.patch, > HBASE-15354-V2.patch, HBASE-15354-V3.patch, HBASE-15354-V4.patch, > HBASE-15354-V5.patch, HBASE-15354-v5-addendum.patch > > > Currently we do not clear/update meta cache for some special exceptions if > the operation went through AsyncProcess#submit like HTable#put calls. But, we > clear meta cache without checking for these special exceptions in case of > other operations like gets, deletes etc because they directly go through the > RpcRetryingCaller#callWithRetries instead of the AsyncProcess. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15413) Procedure-V2: print out ProcedureInfo during trace
[ https://issues.apache.org/jira/browse/HBASE-15413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184587#comment-15184587 ] Hudson commented on HBASE-15413: FAILURE: Integrated in HBase-1.1-JDK7 #1675 (See [https://builds.apache.org/job/HBase-1.1-JDK7/1675/]) HBASE-15413 Procedure-V2: print out ProcedureInfo during trace (Stephen (syuanjiangdev: rev f88df7a4667eee33dbf58579d7f87efc3a99947f) * hbase-common/src/main/java/org/apache/hadoop/hbase/ProcedureInfo.java > Procedure-V2: print out ProcedureInfo during trace > -- > > Key: HBASE-15413 > URL: https://issues.apache.org/jira/browse/HBASE-15413 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Affects Versions: 1.2.0, 1.1.4 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang >Priority: Trivial > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.4.0 > > Attachments: HBASE-15413.v1-master.patch > > > Before the HBASE-15100 refactored the code to print the ProcedureInfo object, > we don't have the needs. Now we need to implement a customized toString() > function to print out information in the ProcedureInfo object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15261) Make Throwable t in DaughterOpener volatile
[ https://issues.apache.org/jira/browse/HBASE-15261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184588#comment-15184588 ] Hudson commented on HBASE-15261: FAILURE: Integrated in HBase-1.1-JDK7 #1675 (See [https://builds.apache.org/job/HBase-1.1-JDK7/1675/]) HBASE-15261 Make Throwable t in DaughterOpener volatile (Huaxiang Sun) (stack: rev 67ec0983b742530571427899404dc391ff4c12e8) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransactionImpl.java > Make Throwable t in DaughterOpener volatile > --- > > Key: HBASE-15261 > URL: https://issues.apache.org/jira/browse/HBASE-15261 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.4.0 > > Attachments: HBASE-15261-001.patch, HBASE-15261-001.patch > > > In the region split process, daughter regions are opened in different > threads, Throwable t is set in these threads and it is checked in the calling > thread. Need to make it volatile so the checking will not miss any exceptions > from opening daughter regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15413) Procedure-V2: print out ProcedureInfo during trace
[ https://issues.apache.org/jira/browse/HBASE-15413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184591#comment-15184591 ] Hudson commented on HBASE-15413: FAILURE: Integrated in HBase-Trunk_matrix #764 (See [https://builds.apache.org/job/HBase-Trunk_matrix/764/]) HBASE-15413 Procedure-V2: print out ProcedureInfo during trace (Stephen (syuanjiangdev: rev 1c6beb3dc1a651e8d40ac0e9c7191727cf8cf413) * hbase-common/src/main/java/org/apache/hadoop/hbase/ProcedureInfo.java > Procedure-V2: print out ProcedureInfo during trace > -- > > Key: HBASE-15413 > URL: https://issues.apache.org/jira/browse/HBASE-15413 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Affects Versions: 1.2.0, 1.1.4 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang >Priority: Trivial > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.4.0 > > Attachments: HBASE-15413.v1-master.patch > > > Before the HBASE-15100 refactored the code to print the ProcedureInfo object, > we don't have the needs. Now we need to implement a customized toString() > function to print out information in the ProcedureInfo object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15421) Convert TestStoreScanner to junit4 from junit3 and clean up close of scanners
[ https://issues.apache.org/jira/browse/HBASE-15421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184590#comment-15184590 ] Hudson commented on HBASE-15421: FAILURE: Integrated in HBase-Trunk_matrix #764 (See [https://builds.apache.org/job/HBase-Trunk_matrix/764/]) HBASE-15421 Convert TestStoreScanner to junit4 from junit3 and clean up (stack: rev 4f044cf6be191b462ec8b5ca3401e8ca6f956f94) * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java > Convert TestStoreScanner to junit4 from junit3 and clean up close of scanners > - > > Key: HBASE-15421 > URL: https://issues.apache.org/jira/browse/HBASE-15421 > Project: HBase > Issue Type: Sub-task > Components: test >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack >Priority: Minor > Fix For: 2.0.0 > > Attachments: 15421.patch > > > I want to add to this test class but it junit3. Let me convert it over first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15137) CallTimeoutException and CallQueueTooBigException should trigger PFFE
[ https://issues.apache.org/jira/browse/HBASE-15137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184589#comment-15184589 ] Hudson commented on HBASE-15137: FAILURE: Integrated in HBase-Trunk_matrix #764 (See [https://builds.apache.org/job/HBase-Trunk_matrix/764/]) HBASE-15137 CallTimeoutException and CallQueueTooBigException should (antonov: rev 46cc3d4972d8db478c86e9848afed523cb6dc866) * hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/PreemptiveFastFailException.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/FastFailInterceptorContext.java * hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/ClientExceptionsUtil.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFastFail.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/PreemptiveFastFailInterceptor.java > CallTimeoutException and CallQueueTooBigException should trigger PFFE > - > > Key: HBASE-15137 > URL: https://issues.apache.org/jira/browse/HBASE-15137 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Mikhail Antonov > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15137.mantonov.patch, HBASE-15137.patch, > HBASE-15137.v3.patch, HBASE-15137.v4.patch, HBASE-15137.v5.patch > > > If a region server is backed up enough that lots of calls are timing out then > we should think about treating a server as failing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15400) Use DateTieredCompactor for Date Tiered Compaction
[ https://issues.apache.org/jira/browse/HBASE-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Clara Xiong updated HBASE-15400: Description: When we compact, we can output multiple files along the current window boundaries. There are two use cases: 1. Major compaction: We want to output date tiered store files with data older than max age archived in trunks of the window size on the higher tier. 2. Bulk load files and the old file generated by major compaction before upgrading to DTCP. Pros: 1. Restore locality, process versioning, updates and deletes while maintaining the tiered layout. 2. The best way to fix a skewed layout. This work is based on a prototype of DateTieredCompactor from HBASE-15389 and focused on the part to meet needs for these two use cases while supporting others. I have to call out a few design decisions: 1. We only want to output the files along all windows for major compaction. And we want to output multiple files older than max age in the trunks of the maximum tier window size determined by base window size, windows per tier and max age. 2. For minor compaction, we don't want to output too many files, which will remain around because of current restriction of contiguous compaction by seq id. I will only output two files if all the files in the windows are being combined, one for the data within window and the other for the out-of-window tail. If there is any file in the window excluded from compaction, only one file will be output from compaction. When the windows are promoted, the situation of out of order data will gradually improve. For the incoming window, we need to accommodate the case with user-specified future data. 3. We have to pass the boundaries with the list of store file as a complete time snapshot instead of two separate calls because window layout is determined by the time the computation is called. So we will need new type of compaction request. 4. Since we will assign the same seq id for all output files, we need to sort by maxTimestamp subsequently. Right now all compaction policy gets the files sorted for StoreFileManager which sorts by seq id and other criteria. I will use this order for DTCP only, to avoid impacting other compaction policies. 5. We need some cleanup of current design of StoreEngine and CompactionPolicy. was: When we compact, we can output multiple files along the current window boundaries. There are two use cases: 1. Major compaction: We want to output date tiered store files with data older than max age archived in trunks of the window size on the higher tier. 2. Bulk load files and the old file generated by major compaction before upgrading to DTCP. Pros: 1. Restore locality, process versioning, updates and deletes while maintaining the tiered layout. 2. The best way to fix a skewed layout. This work is based on a prototype of DateTieredCompactor from HBASE-15389 and focused on the part to meet need for these two use cases while supporting others. I have to call out a few design decisions: 1. We only want to output the files along all windows for major compaction. And we want to output multiple files older than max age in the trunks of the maximum tier window size determined by base window size, windows per tier and max age. 2. For minor compaction, we don't want to output too many files, which will remain around because of current restriction of contiguous compaction by seq id. I will only output two files if all the files in the windows are being combined, one for the data within window and the other for the out-of-window tail. If there is any file in the window excluded from compaction, only one file will be output from compaction. When the windows are promoted, the situation of out of order data will gradually improve. For the incoming window, we need to accommodate the case with user-specified future data. 3. We have to pass the boundaries with the list of store file as a complete time snapshot instead of two separate calls because window layout is determined by the time the computation is called. So we will need new type of compaction request. 4. Since we will assign the same seq id for all output files, we need to sort by maxTimestamp subsequently. Right now all compaction policy gets the files sorted for StoreFileManager which sorts by seq id and other criteria. I will use this order for DTCP only, to avoid impacting other compaction policies. 5. We need some cleanup of current design of StoreEngine and CompactionPolicy. > Use DateTieredCompactor for Date Tiered Compaction > -- > > Key: HBASE-15400 > URL: https://issues.apache.org/jira/browse/HBASE-15400 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Clara Xiong >Assignee: Clara
[jira] [Commented] (HBASE-15389) Write out multiple files when compaction
[ https://issues.apache.org/jira/browse/HBASE-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184570#comment-15184570 ] Clara Xiong commented on HBASE-15389: - 1. We currently use DTCP with DefaultCompactor, which only output a single file from the file list DTCP picks, so that we can compact within every date-tiered window into a single file and the resulted layout will be tiered by date. We will use this new compactor WITH DTCP to output date tiered layout if the file(s) to compact expands over multiple time windows such as a major compaction or bulk load files. So DTCP is not going away. By pairing it with this new compactor, we can improve for some use cases. HBASE-15400 has more details. 2. The windowing logic in DateTieredMultiFileWriter you referred to is to output data to multiple files based on a given list of boundaries. These boundaries will be determined in DTCP and passed as part of DateTieredCompactionRequest it generates to DateTieredCompactor. > Write out multiple files when compaction > > > Key: HBASE-15389 > URL: https://issues.apache.org/jira/browse/HBASE-15389 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 0.98.19 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.3.0, 0.98.19 > > Attachments: HBASE-15389-uc.patch, HBASE-15389-v1.patch, > HBASE-15389-v2.patch, HBASE-15389.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-15392) Single Cell Get reads two HFileBlocks
[
https://issues.apache.org/jira/browse/HBASE-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184524#comment-15184524
]
Lars Hofhansl edited comment on HBASE-15392 at 3/8/16 6:40 AM:
---
OK I get it now... It's doing the right thing. HConstants.OLDEST_TIMESTAMP,
Type.Minimum.getCode() produces the _last_ KV for the current row.
But note that that would still be the Cell _right before_ the nextIndexedKey
(since that is the first key of the _next_ block), and hence we'll need to
check the next one and overshoop by one Cell (and hence load the next block).
Instead of the first key of the next block, what we'd really want is the last
key of the current block to compare against - but we do not have that readily
available, since we'd have to scan forward in the block to the last Cell in
order to find its key.
So, it seems like -looksee is still the best option.
was (Author: lhofhansl):
OK I get it now... It's doing the right thing. HConstants.OLDEST_TIMESTAMP,
Type.Minimum.getCode() produces the _last_ KV for the current row.
But note that that would still be the Cell _right before_ the nextIndexedKey
(since that is the first key of the _next_ block), and hence we'll need to
check the next one and overshoop by one Cell (and hence load the next block).
Instead of the first key of the next block, what we'd really want is the last
key of the current block to compare against - but we do not have that readily
available.
> Single Cell Get reads two HFileBlocks
> -
>
> Key: HBASE-15392
> URL: https://issues.apache.org/jira/browse/HBASE-15392
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: stack
>Assignee: stack
> Attachments: 15392-0.98-looksee.txt, 15392.wip.patch,
> 15392v2.wip.patch, 15392v3.wip.patch, HBASE-15392_suggest.patch,
> no_optimize.patch, no_optimize.patch, two_seeks.txt
>
>
> As found by Daniel "SystemTap" Pol, a simple Get results in our reading two
> HFileBlocks, the one that contains the wanted Cell, and the block that
> follows.
> Here is a bit of custom logging that logs a stack trace on each HFileBlock
> read so you can see the call stack responsible:
> {code}
> 2016-03-03 22:20:30,191 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] regionserver.StoreScanner:
> START LOOP
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] regionserver.StoreScanner:
> QCODE SEEK_NEXT_COL
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileBlockIndex:
> STARTED WHILE
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.CombinedBlockCache:
> OUT OF L2
> 2016-03-03 22:20:30,192 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] bucket.BucketCache: Read
> offset=31409152, len=2103
> 2016-03-03 22:20:30,192 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] bucket.FileIOEngine:
> offset=31409152, length=2103
> 2016-03-03 22:20:30,193 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileReaderImpl:
> From Cache [blockType=DATA, fileOffset=2055421, headerSize=33,
> onDiskSizeWithoutHeader=2024, uncompressedSizeWithoutHeader=2020,
> prevBlockOffset=2053364, isUseHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, onDiskDataSizeWithHeader=2053,
> getOnDiskSizeWithHeader=2057, totalChecksumBytes=4, isUnpacked=true,
> buf=[org.apache.hadoop.hbase.nio.SingleByteBuff@e19fbd54],
> dataBeginsWith=\x00\x00\x00)\x00\x00\x01`\x00\x16user995139035672819231,
> fileContext=[usesHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true,
> includesTags=false, compressAlgo=NONE, compressTags=false,
> cryptoContext=[cipher=NONE keyHash=NONE]]]
> 2016-03-03 22:20:30,193 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileReaderImpl:
> Cache hit return [blockType=DATA, fileOffset=2055421, headerSize=33,
> onDiskSizeWithoutHeader=2024, uncompressedSizeWithoutHeader=2020,
> prevBlockOffset=2053364, isUseHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, onDiskDataSizeWithHeader=2053,
> getOnDiskSizeWithHeader=2057, totalChecksumBytes=4, isUnpacked=true,
> buf=[org.apache.hadoop.hbase.nio.SingleByteBuff@e19fbd54],
> dataBeginsWith=\x00\x00\x00)\x00\x00\x01`\x00\x16user995139035672819231,
> fileContext=[usesHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true,
> includesTags=false, compressAlgo=NONE, compressTags=false,
> cryptoContext=[cipher=NONE keyHash=NONE]]]
> java.lang.Throwable
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1515)
>
[jira] [Commented] (HBASE-15422) Procedure v2 - Avoid double yield
[ https://issues.apache.org/jira/browse/HBASE-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184526#comment-15184526 ] Stephen Yuan Jiang commented on HBASE-15422: +1 LGTM > Procedure v2 - Avoid double yield > - > > Key: HBASE-15422 > URL: https://issues.apache.org/jira/browse/HBASE-15422 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 1.2.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15422-v0.patch > > > ServerCrashProcedure is using a combination of > isYieldBeforeExecuteFromState() and ProcedureYieldException, which may end up > in yielding twice. (ServerCrashProcedure is the only user of yield) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15422) Procedure v2 - Avoid double yield
[
https://issues.apache.org/jira/browse/HBASE-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184525#comment-15184525
]
Hadoop QA commented on HBASE-15422:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 12s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 45s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 51s
{color} | {color:green} hbase-procedure in the patch passed with JDK v1.8.0.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 47s
{color} | {color:green} hbase-procedure in the patch passed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 8s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12791936/HBASE-15422-v0.patch |
| JIRA Issue | HBASE-15422 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux asf910.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 46cc3d4 |
| Default Java | 1.7.0_79 |
| Multi-JDK versions | /home/jenkins/tools/java
[jira] [Commented] (HBASE-15392) Single Cell Get reads two HFileBlocks
[
https://issues.apache.org/jira/browse/HBASE-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184524#comment-15184524
]
Lars Hofhansl commented on HBASE-15392:
---
OK I get it now... It's doing the right thing. HConstants.OLDEST_TIMESTAMP,
Type.Minimum.getCode() produces the _last_ KV for the current row.
But note that that would still be the Cell _right before_ the nextIndexedKey
(since that is the first key of the _next_ block), and hence we'll need to
check the next one and overshoop by one Cell (and hence load the next block).
Instead of the first key of the next block, what we'd really want is the last
key of the current block to compare against - but we do not have that readily
available.
> Single Cell Get reads two HFileBlocks
> -
>
> Key: HBASE-15392
> URL: https://issues.apache.org/jira/browse/HBASE-15392
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: stack
>Assignee: stack
> Attachments: 15392-0.98-looksee.txt, 15392.wip.patch,
> 15392v2.wip.patch, 15392v3.wip.patch, HBASE-15392_suggest.patch,
> no_optimize.patch, no_optimize.patch, two_seeks.txt
>
>
> As found by Daniel "SystemTap" Pol, a simple Get results in our reading two
> HFileBlocks, the one that contains the wanted Cell, and the block that
> follows.
> Here is a bit of custom logging that logs a stack trace on each HFileBlock
> read so you can see the call stack responsible:
> {code}
> 2016-03-03 22:20:30,191 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] regionserver.StoreScanner:
> START LOOP
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] regionserver.StoreScanner:
> QCODE SEEK_NEXT_COL
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileBlockIndex:
> STARTED WHILE
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.CombinedBlockCache:
> OUT OF L2
> 2016-03-03 22:20:30,192 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] bucket.BucketCache: Read
> offset=31409152, len=2103
> 2016-03-03 22:20:30,192 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] bucket.FileIOEngine:
> offset=31409152, length=2103
> 2016-03-03 22:20:30,193 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileReaderImpl:
> From Cache [blockType=DATA, fileOffset=2055421, headerSize=33,
> onDiskSizeWithoutHeader=2024, uncompressedSizeWithoutHeader=2020,
> prevBlockOffset=2053364, isUseHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, onDiskDataSizeWithHeader=2053,
> getOnDiskSizeWithHeader=2057, totalChecksumBytes=4, isUnpacked=true,
> buf=[org.apache.hadoop.hbase.nio.SingleByteBuff@e19fbd54],
> dataBeginsWith=\x00\x00\x00)\x00\x00\x01`\x00\x16user995139035672819231,
> fileContext=[usesHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true,
> includesTags=false, compressAlgo=NONE, compressTags=false,
> cryptoContext=[cipher=NONE keyHash=NONE]]]
> 2016-03-03 22:20:30,193 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileReaderImpl:
> Cache hit return [blockType=DATA, fileOffset=2055421, headerSize=33,
> onDiskSizeWithoutHeader=2024, uncompressedSizeWithoutHeader=2020,
> prevBlockOffset=2053364, isUseHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, onDiskDataSizeWithHeader=2053,
> getOnDiskSizeWithHeader=2057, totalChecksumBytes=4, isUnpacked=true,
> buf=[org.apache.hadoop.hbase.nio.SingleByteBuff@e19fbd54],
> dataBeginsWith=\x00\x00\x00)\x00\x00\x01`\x00\x16user995139035672819231,
> fileContext=[usesHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true,
> includesTags=false, compressAlgo=NONE, compressTags=false,
> cryptoContext=[cipher=NONE keyHash=NONE]]]
> java.lang.Throwable
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1515)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$CellBasedKeyBlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:324)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:831)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.reseekTo(HFileReaderImpl.java:812)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:288)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:198)
> at
> org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54)
> at
> org.apache.hadoop.hbase.regionserver.K
[jira] [Commented] (HBASE-15265) Implement an asynchronous FSHLog
[
https://issues.apache.org/jira/browse/HBASE-15265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184500#comment-15184500
]
Duo Zhang commented on HBASE-15265:
---
Run WALPE with 5 threads
{noformat}
./bin/hbase org.apache.hadoop.hbase.wal.WALPerformanceEvaluation -threads 5
{noformat}
master
{noformat}
-- Histograms --
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.latencyHistogram.nanos
count = 500
min = 452200
max = 1834676
mean = 723087.74
stddev = 125994.96
median = 703103.00
75% <= 745173.00
95% <= 920767.00
98% <= 1148129.00
99% <= 1288697.00
99.9% <= 1554315.00
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.syncCountHistogram.countPerSync
count = 4571938
min = 1
max = 2
mean = 1.06
stddev = 0.23
median = 1.00
75% <= 1.00
95% <= 2.00
98% <= 2.00
99% <= 2.00
99.9% <= 2.00
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.syncHistogram.nanos-between-syncs
count = 4571938
min = 415874
max = 1571407
mean = 693891.09
stddev = 107159.47
median = 673660.00
75% <= 717641.00
95% <= 896732.00
98% <= 992794.00
99% <= 1098338.00
99.9% <= 1518093.00
-- Meters --
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.appendMeter.bytes
count = 278500
mean rate = 3354500.98 events/second
1-minute rate = 3613345.23 events/second
5-minute rate = 3254224.23 events/second
15-minute rate = 2049616.71 events/second
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.syncMeter.syncs
count = 4571938
mean rate = 5506.81 events/second
1-minute rate = 6090.84 events/second
5-minute rate = 5397.76 events/second
15-minute rate = 3376.47 events/second
2016-03-08 13:59:10,692 INFO [t4,r0] wal.WALPerformanceEvaluation: t4,r0 took
809.430s 1235.437ops/s
2016-03-08 13:59:10,730 INFO [t0,r0] wal.WALPerformanceEvaluation: t0,r0 took
809.469s 1235.378ops/s
2016-03-08 13:59:10,925 INFO [t3,r0] wal.WALPerformanceEvaluation: t3,r0 took
809.664s 1235.080ops/s
2016-03-08 13:59:11,164 INFO [t2,r0] wal.WALPerformanceEvaluation: t2,r0 took
809.903s 1234.716ops/s
2016-03-08 13:59:11,232 INFO [t1,r0] wal.WALPerformanceEvaluation: t1,r0 took
809.972s 1234.611ops/s
2016-03-08 13:59:11,233 INFO [main] wal.WALPerformanceEvaluation: Summary:
threads=5, iterations=100, syncInterval=0 took 809.973s 6173.045ops/s
{noformat}
patched
{noformat}
-- Histograms --
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.latencyHistogram.nanos
count = 500
min = 67653
max = 1211514
mean = 355929.64
stddev = 100616.20
median = 338680.00
75% <= 386992.00
95% <= 466571.00
98% <= 678368.00
99% <= 767800.00
99.9% <= 1174408.00
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.syncCountHistogram.countPerSync
count = 2982546
min = 0
max = 4
mean = 1.63
stddev = 0.74
median = 2.00
75% <= 2.00
95% <= 3.00
98% <= 4.00
99% <= 4.00
99.9% <= 4.00
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.syncHistogram.nanos-between-syncs
count = 2982546
min = 180697
max = 57874230
mean = 375459.18
stddev = 1551439.38
median = 310104.00
75% <= 349804.00
95% <= 469386.00
98% <= 755095.00
99% <= 1064668.00
99.9% <= 1679322.00
-- Meters --
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.appendMeter.bytes
count = 278500
mean rate = 6784702.13 events/second
1-minute rate = 6957392.69 events/second
5-minute rate = 5188375.04 events/second
15-minute rate = 2511006.30 events/second
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.syncMeter.syncs
count = 2982546
mean rate = 7265.86 events/second
1-minute rate = 7499.49 events/second
5-minute rate = 5574.20 events/second
15-minute rate = 2692.50 events/second
2016-03-08 14:06:59,454 INFO [t3,r0] wal.WALPerformanceEvalua
[jira] [Updated] (HBASE-15422) Procedure v2 - Avoid double yield
[ https://issues.apache.org/jira/browse/HBASE-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-15422: Issue Type: Sub-task (was: Bug) Parent: HBASE-14336 > Procedure v2 - Avoid double yield > - > > Key: HBASE-15422 > URL: https://issues.apache.org/jira/browse/HBASE-15422 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 1.2.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15422-v0.patch > > > ServerCrashProcedure is using a combination of > isYieldBeforeExecuteFromState() and ProcedureYieldException, which may end up > in yielding twice. (ServerCrashProcedure is the only user of yield) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15416) TestHFileBackedByBucketCache is flakey since it went in
[ https://issues.apache.org/jira/browse/HBASE-15416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184451#comment-15184451 ] Hudson commented on HBASE-15416: FAILURE: Integrated in HBase-Trunk_matrix #763 (See [https://builds.apache.org/job/HBase-Trunk_matrix/763/]) HBASE-15416 TestHFileBackedByBucketCache is flakey since it went in (stack: rev 0f14856b01ed14eca9b1cd91087d85732b3ad2d3) * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBackedByBucketCache.java > TestHFileBackedByBucketCache is flakey since it went in > --- > > Key: HBASE-15416 > URL: https://issues.apache.org/jira/browse/HBASE-15416 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: 15416.patch > > > Looks like cache content changes during running of test... let me fix. > Critical. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15354) Use same criteria for clearing meta cache for all operations
[ https://issues.apache.org/jira/browse/HBASE-15354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184450#comment-15184450 ] Hudson commented on HBASE-15354: FAILURE: Integrated in HBase-Trunk_matrix #763 (See [https://builds.apache.org/job/HBase-Trunk_matrix/763/]) HBASE-15354 Use same criteria for clearing meta cache for all operations (antonov: rev e477c143bca83b8ec8e10c65a0e8ef526e24482e) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMetaCache.java > Use same criteria for clearing meta cache for all operations > > > Key: HBASE-15354 > URL: https://issues.apache.org/jira/browse/HBASE-15354 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 1.2.1 > > Attachments: HBASE-15354-V0.patch, HBASE-15354-V1.patch, > HBASE-15354-V2.patch, HBASE-15354-V3.patch, HBASE-15354-V4.patch, > HBASE-15354-V5.patch, HBASE-15354-v5-addendum.patch > > > Currently we do not clear/update meta cache for some special exceptions if > the operation went through AsyncProcess#submit like HTable#put calls. But, we > clear meta cache without checking for these special exceptions in case of > other operations like gets, deletes etc because they directly go through the > RpcRetryingCaller#callWithRetries instead of the AsyncProcess. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15261) Make Throwable t in DaughterOpener volatile
[ https://issues.apache.org/jira/browse/HBASE-15261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184452#comment-15184452 ] Hudson commented on HBASE-15261: FAILURE: Integrated in HBase-Trunk_matrix #763 (See [https://builds.apache.org/job/HBase-Trunk_matrix/763/]) HBASE-15261 Make Throwable t in DaughterOpener volatile (Huaxiang Sun) (stack: rev d15ae0e6abea3e25838df6f8850c9593248fe6a4) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransactionImpl.java > Make Throwable t in DaughterOpener volatile > --- > > Key: HBASE-15261 > URL: https://issues.apache.org/jira/browse/HBASE-15261 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.4.0 > > Attachments: HBASE-15261-001.patch, HBASE-15261-001.patch > > > In the region split process, daughter regions are opened in different > threads, Throwable t is set in these threads and it is checked in the calling > thread. Need to make it volatile so the checking will not miss any exceptions > from opening daughter regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15392) Single Cell Get reads two HFileBlocks
[
https://issues.apache.org/jira/browse/HBASE-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184448#comment-15184448
]
Lars Hofhansl commented on HBASE-15392:
---
Actually here's another thought. What optimize is trying to figure out is
whether the SEEK would get us onto a new HFileBlock, if so, it'll do the SEEK.
So the description is _exactly_ what we want: If we'd SEEK onto the next block
it would leave the SEEK in place.
So why isn't it working? In SQM.compareKeyForNextRow I compare the next indexed
key with HConstants.OLDEST_TIMESTAMP, Type.Minimum.getCode(). Seems for the
purpose of this optimization we should instead use HConstants.LATEST_TIMESTAMP,
Type.Maximum.getCode(), that way the indexedKey of the next block would _after_
this key, not before.
I still cannot reproduce this easily. So I'll produce a new patch in minute.
Let me know what you think.
> Single Cell Get reads two HFileBlocks
> -
>
> Key: HBASE-15392
> URL: https://issues.apache.org/jira/browse/HBASE-15392
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: stack
>Assignee: stack
> Attachments: 15392-0.98-looksee.txt, 15392.wip.patch,
> 15392v2.wip.patch, 15392v3.wip.patch, HBASE-15392_suggest.patch,
> no_optimize.patch, no_optimize.patch, two_seeks.txt
>
>
> As found by Daniel "SystemTap" Pol, a simple Get results in our reading two
> HFileBlocks, the one that contains the wanted Cell, and the block that
> follows.
> Here is a bit of custom logging that logs a stack trace on each HFileBlock
> read so you can see the call stack responsible:
> {code}
> 2016-03-03 22:20:30,191 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] regionserver.StoreScanner:
> START LOOP
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] regionserver.StoreScanner:
> QCODE SEEK_NEXT_COL
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileBlockIndex:
> STARTED WHILE
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.CombinedBlockCache:
> OUT OF L2
> 2016-03-03 22:20:30,192 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] bucket.BucketCache: Read
> offset=31409152, len=2103
> 2016-03-03 22:20:30,192 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] bucket.FileIOEngine:
> offset=31409152, length=2103
> 2016-03-03 22:20:30,193 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileReaderImpl:
> From Cache [blockType=DATA, fileOffset=2055421, headerSize=33,
> onDiskSizeWithoutHeader=2024, uncompressedSizeWithoutHeader=2020,
> prevBlockOffset=2053364, isUseHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, onDiskDataSizeWithHeader=2053,
> getOnDiskSizeWithHeader=2057, totalChecksumBytes=4, isUnpacked=true,
> buf=[org.apache.hadoop.hbase.nio.SingleByteBuff@e19fbd54],
> dataBeginsWith=\x00\x00\x00)\x00\x00\x01`\x00\x16user995139035672819231,
> fileContext=[usesHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true,
> includesTags=false, compressAlgo=NONE, compressTags=false,
> cryptoContext=[cipher=NONE keyHash=NONE]]]
> 2016-03-03 22:20:30,193 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileReaderImpl:
> Cache hit return [blockType=DATA, fileOffset=2055421, headerSize=33,
> onDiskSizeWithoutHeader=2024, uncompressedSizeWithoutHeader=2020,
> prevBlockOffset=2053364, isUseHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, onDiskDataSizeWithHeader=2053,
> getOnDiskSizeWithHeader=2057, totalChecksumBytes=4, isUnpacked=true,
> buf=[org.apache.hadoop.hbase.nio.SingleByteBuff@e19fbd54],
> dataBeginsWith=\x00\x00\x00)\x00\x00\x01`\x00\x16user995139035672819231,
> fileContext=[usesHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true,
> includesTags=false, compressAlgo=NONE, compressTags=false,
> cryptoContext=[cipher=NONE keyHash=NONE]]]
> java.lang.Throwable
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1515)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$CellBasedKeyBlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:324)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:831)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.reseekTo(HFileReaderImpl.java:812)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:288)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:198)
>
[jira] [Commented] (HBASE-15265) Implement an asynchronous FSHLog
[ https://issues.apache.org/jira/browse/HBASE-15265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184446#comment-15184446 ] stack commented on HBASE-15265: --- Excellent. I owe review and a run of ITBLL up on cluster to see that all basically works. Please excuse me. I am a little backed up and cluster is doing other tests at mo. Should be cleared in next day or so. Thanks @duo. > Implement an asynchronous FSHLog > > > Key: HBASE-15265 > URL: https://issues.apache.org/jira/browse/HBASE-15265 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15265-v1.patch, HBASE-15265-v2.patch, > HBASE-15265-v3.patch, HBASE-15265-v4.patch, HBASE-15265-v5.patch, > HBASE-15265-v6.patch, HBASE-15265.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184443#comment-15184443 ] Amal Joshy commented on HBASE-15314: Submitted initial patch. Please review. > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Amal Joshy > Attachments: HBASE-15314.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amal Joshy updated HBASE-15314: --- Attachment: HBASE-15314.patch > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Amal Joshy > Attachments: HBASE-15314.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15421) Convert TestStoreScanner to junit4 from junit3 and clean up close of scanners
[ https://issues.apache.org/jira/browse/HBASE-15421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15421: -- Priority: Minor (was: Major) > Convert TestStoreScanner to junit4 from junit3 and clean up close of scanners > - > > Key: HBASE-15421 > URL: https://issues.apache.org/jira/browse/HBASE-15421 > Project: HBase > Issue Type: Sub-task > Components: test >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack >Priority: Minor > Fix For: 2.0.0 > > Attachments: 15421.patch > > > I want to add to this test class but it junit3. Let me convert it over first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-15421) Convert TestStoreScanner to junit4 from junit3 and clean up close of scanners
[ https://issues.apache.org/jira/browse/HBASE-15421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-15421. --- Resolution: Fixed Fix Version/s: 2.0.0 Resolving. Simple refactoring of a test. > Convert TestStoreScanner to junit4 from junit3 and clean up close of scanners > - > > Key: HBASE-15421 > URL: https://issues.apache.org/jira/browse/HBASE-15421 > Project: HBase > Issue Type: Sub-task > Components: test >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: 15421.patch > > > I want to add to this test class but it junit3. Let me convert it over first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15265) Implement an asynchronous FSHLog
[
https://issues.apache.org/jira/browse/HBASE-15265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184421#comment-15184421
]
Duo Zhang commented on HBASE-15265:
---
[~stack] I have run a WALPE on a 5 node cluster.
On master branch
{noformat}
-- Histograms --
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.latencyHistogram.nanos
count = 993526
min = 424172
max = 1376267
mean = 491388.58
stddev = 54001.33
median = 483846.00
75% <= 496580.00
95% <= 545363.00
98% <= 588029.00
99% <= 645355.00
99.9% <= 1376267.00
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.syncCountHistogram.countPerSync
count = 993538
min = 1
max = 1
mean = 1.00
stddev = 0.00
median = 1.00
75% <= 1.00
95% <= 1.00
98% <= 1.00
99% <= 1.00
99.9% <= 1.00
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.syncHistogram.nanos-between-syncs
count = 993540
min = 390501
max = 1200319
mean = 466742.40
stddev = 41737.18
median = 460329.00
75% <= 473633.00
95% <= 516878.00
98% <= 556977.00
99% <= 610978.00
99.9% <= 821039.00
-- Meters --
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.appendMeter.bytes
count = 553402337
mean rate = 1045989.09 events/second
1-minute rate = 1087437.57 events/second
5-minute rate = 879784.11 events/second
15-minute rate = 465526.36 events/second
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.syncMeter.syncs
count = 993546
mean rate = 1877.89 events/second
1-minute rate = 1952.33 events/second
5-minute rate = 1579.51 events/second
15-minute rate = 835.77 events/second
2016-03-08 12:32:15,664 INFO [t0,r0] wal.WALPerformanceEvaluation: t0,r0 took
513.158s 1948.718ops/s
2016-03-08 12:32:15,664 INFO [main] wal.WALPerformanceEvaluation: Summary:
threads=1, iterations=100, syncInterval=0 took 513.158s 1948.718ops/s
{noformat}
With the v6 patch and set {{hbase.wal.provider}} to {{asyncfs}}
{noformat}
-- Histograms --
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.latencyHistogram.nanos
count = 942227
min = 178089
max = 16710130
mean = 274607.68
stddev = 707975.65
median = 240741.00
75% <= 251231.00
95% <= 283781.00
98% <= 337094.00
99% <= 460342.00
99.9% <= 16710130.00
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.syncCountHistogram.countPerSync
count = 942235
min = 1
max = 1
mean = 1.00
stddev = 0.00
median = 1.00
75% <= 1.00
95% <= 1.00
98% <= 1.00
99% <= 1.00
99.9% <= 1.00
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.syncHistogram.nanos-between-syncs
count = 942241
min = 162382
max = 6048183
mean = 231193.13
stddev = 231598.71
median = 220296.00
75% <= 228857.00
95% <= 264478.00
98% <= 302176.00
99% <= 398986.00
99.9% <= 6048183.00
-- Meters --
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.appendMeter.bytes
count = 524833250
mean rate = 2011339.83 events/second
1-minute rate = 2168110.13 events/second
5-minute rate = 1203438.51 events/second
15-minute rate = 510366.96 events/second
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.syncMeter.syncs
count = 942251
mean rate = 3610.95 events/second
1-minute rate = 3892.46 events/second
5-minute rate = 2160.54 events/second
15-minute rate = 916.26 events/second
2016-03-08 12:22:14,850 INFO [t0,r0] wal.WALPerformanceEvaluation: t0,r0 took
254.562s 3928.316ops/s
2016-03-08 12:22:14,850 INFO [main] wal.WALPerformanceEvaluation: Summary:
threads=1, iterations=100, syncInterval=0 took 254.562s 3928.316ops/s
{noformat}
Almost the same with your previous test, about 2x faster for singie threaded
test.
3928.316ops/s vs. 1948.718ops/s
> Implement an asynchronous FSHLog
>
>
> Key: HBASE-15265
>
[jira] [Updated] (HBASE-15413) Procedure-V2: print out ProcedureInfo during trace
[ https://issues.apache.org/jira/browse/HBASE-15413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-15413: --- Resolution: Fixed Fix Version/s: 1.4.0 1.1.4 1.2.1 1.3.0 2.0.0 Status: Resolved (was: Patch Available) > Procedure-V2: print out ProcedureInfo during trace > -- > > Key: HBASE-15413 > URL: https://issues.apache.org/jira/browse/HBASE-15413 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Affects Versions: 1.2.0, 1.1.4 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang >Priority: Trivial > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.4.0 > > Attachments: HBASE-15413.v1-master.patch > > > Before the HBASE-15100 refactored the code to print the ProcedureInfo object, > we don't have the needs. Now we need to implement a customized toString() > function to print out information in the ProcedureInfo object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15322) HBase 1.1.3 crashing
[ https://issues.apache.org/jira/browse/HBASE-15322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184417#comment-15184417 ] Hiroshi Ikeda commented on HBASE-15322: --- In actual, that depends on VM implementations. VM can load a class anywhere VM find its necessity, but in practice it seems that just setting or referring null instance doesn't trigger to load the class. I think calling a static/instance method causes loading even if there is just null reference. It is the best to cut off the dependency to the implementation class, like UnsafeComparer - that was quite well encapsulated and I feel the huge effort the developer invested behind the code, but that is spoiled at least in 1.1.3. > HBase 1.1.3 crashing > > > Key: HBASE-15322 > URL: https://issues.apache.org/jira/browse/HBASE-15322 > Project: HBase > Issue Type: Bug > Components: hbase >Affects Versions: 1.0.0, 2.0.0, 0.98.7, 0.94.24 > Environment: OS: Ubuntu 14.04/Ubuntu 15.10 > JDK: OpenJDK8/OpenJDK9 >Reporter: Anant Sharma >Assignee: Anoop Sam John >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 0.98.18, 1.4.0 > > Attachments: BASE-15322.patch > > > HBase crashes in standalone mode with the following log: > __ > 2016-02-24 22:38:37,578 ERROR [main] master.HMasterCommandLine: Master exiting > java.lang.RuntimeException: Failed construction of Master: class > org.apache.hadoop.hbase.master.HMaster > at > org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2341) > at > org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:233) > at > org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126) > at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2355) > Caused by: java.lang.NoClassDefFoundError: Could not initialize class > org.apache.hadoop.hbase.util.Bytes$LexicographicalComparerHolder$UnsafeComparer > at org.apache.hadoop.hbase.util.Bytes.putInt(Bytes.java:899) > at > org.apache.hadoop.hbase.KeyValue.createByteArray(KeyValue.java:1082) > at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:652) > at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:580) > at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:483) > at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:370) > at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:267) > at org.apache.hadoop.hbase.HConstants.(HConstants.java:978) > at > org.apache.hadoop.hbase.HTableDescriptor.(HTableDescriptor.java:1488) > at > org.apache.hadoop.hbase.util.FSTableDescriptors.(FSTableDescriptors.java:124) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:570) > at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:365) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2336) > __ > The class is in the hbase-common.jar and its there in the classpath as can be > seen from the log: > _ > 2016-02-24 22:38:32,538 INFO [main] util.ServerCommandLine: > env:CLASSPATH=/home/hduser/hbase/hbase-1.1.3:/home/hduser/hbase/hbase-1.1.3/lib/activation-1.1.jar:/home/hduser/hbase/hbase-1.1.3/lib/aopalliance-1.0.jar:/home/hduser/hbase/hbase-1.1.3/lib/apacheds-i18n-2.0.0-M15.jar:/home/hduser/hbase/hbase-1.1.3/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hduser/hbase/hbase-1.1.3/lib/api-asn1-api-1.0.0-M20.jar:/home/hduser/hbase/hbase-1.1.3/lib/api-util-1.0.0-M20.jar:/home/hduser/hbase/hbase-1.1.3/lib/asm-3.1.jar:/home/hduser/hbase/hbase-1.1.3/lib/avro-1.7.4.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-beanutils-1.7.0.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-beanutils-core-1.8.0.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-cli-1.2.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-codec-1.9.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-collections-3.2.2.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-compress-1.4.1.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-configuration-1.6.jar
[jira] [Updated] (HBASE-15422) Procedure v2 - Avoid double yield
[ https://issues.apache.org/jira/browse/HBASE-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-15422: Status: Patch Available (was: Open) > Procedure v2 - Avoid double yield > - > > Key: HBASE-15422 > URL: https://issues.apache.org/jira/browse/HBASE-15422 > Project: HBase > Issue Type: Bug > Components: proc-v2 >Affects Versions: 1.2.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15422-v0.patch > > > ServerCrashProcedure is using a combination of > isYieldBeforeExecuteFromState() and ProcedureYieldException, which may end up > in yielding twice. (ServerCrashProcedure is the only user of yield) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15422) Procedure v2 - Avoid double yield
[ https://issues.apache.org/jira/browse/HBASE-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-15422: Attachment: HBASE-15422-v0.patch > Procedure v2 - Avoid double yield > - > > Key: HBASE-15422 > URL: https://issues.apache.org/jira/browse/HBASE-15422 > Project: HBase > Issue Type: Bug > Components: proc-v2 >Affects Versions: 1.2.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15422-v0.patch > > > ServerCrashProcedure is using a combination of > isYieldBeforeExecuteFromState() and ProcedureYieldException, which may end up > in yielding twice. (ServerCrashProcedure is the only user of yield) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15422) Procedure v2 - Avoid double yield
[ https://issues.apache.org/jira/browse/HBASE-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-15422: Attachment: (was: HBASE-15422-v0.patch) > Procedure v2 - Avoid double yield > - > > Key: HBASE-15422 > URL: https://issues.apache.org/jira/browse/HBASE-15422 > Project: HBase > Issue Type: Bug > Components: proc-v2 >Affects Versions: 1.2.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15422-v0.patch > > > ServerCrashProcedure is using a combination of > isYieldBeforeExecuteFromState() and ProcedureYieldException, which may end up > in yielding twice. (ServerCrashProcedure is the only user of yield) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15422) Procedure v2 - Avoid double yield
[ https://issues.apache.org/jira/browse/HBASE-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-15422: Attachment: HBASE-15422-v0.patch > Procedure v2 - Avoid double yield > - > > Key: HBASE-15422 > URL: https://issues.apache.org/jira/browse/HBASE-15422 > Project: HBase > Issue Type: Bug > Components: proc-v2 >Affects Versions: 1.2.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15422-v0.patch > > > ServerCrashProcedure is using a combination of > isYieldBeforeExecuteFromState() and ProcedureYieldException, which may end up > in yielding twice. (ServerCrashProcedure is the only user of yield) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15422) Procedure v2 - Avoid double yield
Matteo Bertozzi created HBASE-15422: --- Summary: Procedure v2 - Avoid double yield Key: HBASE-15422 URL: https://issues.apache.org/jira/browse/HBASE-15422 Project: HBase Issue Type: Bug Components: proc-v2 Affects Versions: 1.2.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 2.0.0, 1.3.0, 1.2.1 ServerCrashProcedure is using a combination of isYieldBeforeExecuteFromState() and ProcedureYieldException, which may end up in yielding twice. (ServerCrashProcedure is the only user of yield) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15265) Implement an asynchronous FSHLog
[ https://issues.apache.org/jira/browse/HBASE-15265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-15265: -- Attachment: HBASE-15265-v6.patch Fix a typo that cause negative value of metrics. > Implement an asynchronous FSHLog > > > Key: HBASE-15265 > URL: https://issues.apache.org/jira/browse/HBASE-15265 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15265-v1.patch, HBASE-15265-v2.patch, > HBASE-15265-v3.patch, HBASE-15265-v4.patch, HBASE-15265-v5.patch, > HBASE-15265-v6.patch, HBASE-15265.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15413) Procedure-V2: print out ProcedureInfo during trace
[ https://issues.apache.org/jira/browse/HBASE-15413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184396#comment-15184396 ] Stephen Yuan Jiang commented on HBASE-15413: This change does not need unit test. The toString() function would be called by a lot of unit tests that needs to trace the ProcedureInfo object > Procedure-V2: print out ProcedureInfo during trace > -- > > Key: HBASE-15413 > URL: https://issues.apache.org/jira/browse/HBASE-15413 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Affects Versions: 1.2.0, 1.1.4 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang >Priority: Trivial > Attachments: HBASE-15413.v1-master.patch > > > Before the HBASE-15100 refactored the code to print the ProcedureInfo object, > we don't have the needs. Now we need to implement a customized toString() > function to print out information in the ProcedureInfo object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15421) Convert TestStoreScanner to junit4 from junit3 and clean up close of scanners
[ https://issues.apache.org/jira/browse/HBASE-15421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15421: -- Affects Version/s: 2.0.0 Dang. Can only push to master w/o work. That'll do for now. > Convert TestStoreScanner to junit4 from junit3 and clean up close of scanners > - > > Key: HBASE-15421 > URL: https://issues.apache.org/jira/browse/HBASE-15421 > Project: HBase > Issue Type: Sub-task > Components: test >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack > Attachments: 15421.patch > > > I want to add to this test class but it junit3. Let me convert it over first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15421) Convert TestStoreScanner to junit4 from junit3 and clean up close of scanners
[ https://issues.apache.org/jira/browse/HBASE-15421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15421: -- Attachment: 15421.patch What I pushed. > Convert TestStoreScanner to junit4 from junit3 and clean up close of scanners > - > > Key: HBASE-15421 > URL: https://issues.apache.org/jira/browse/HBASE-15421 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: stack > Attachments: 15421.patch > > > I want to add to this test class but it junit3. Let me convert it over first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15421) Convert TestStoreScanner to junit4 from junit3 and clean up close of scanners
stack created HBASE-15421: - Summary: Convert TestStoreScanner to junit4 from junit3 and clean up close of scanners Key: HBASE-15421 URL: https://issues.apache.org/jira/browse/HBASE-15421 Project: HBase Issue Type: Sub-task Components: test Reporter: stack Assignee: stack I want to add to this test class but it junit3. Let me convert it over first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15392) Single Cell Get reads two HFileBlocks
[
https://issues.apache.org/jira/browse/HBASE-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184389#comment-15184389
]
ramkrishna.s.vasudevan commented on HBASE-15392:
[~lhofhansl]
I think it is ok to do only for gets. Suppose we have a scan with few columns
specified we will get this INCLUDE_SEEK_NEXT_ROW and that will always have to
do a check if we do moreRowsMayExistAfter() for scans also. (ie for every ROW).
My 'suggest' patch was doing more compares in such cases but I think your
optimize logic is more to avoid the seek unnecessarily.
> Single Cell Get reads two HFileBlocks
> -
>
> Key: HBASE-15392
> URL: https://issues.apache.org/jira/browse/HBASE-15392
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: stack
>Assignee: stack
> Attachments: 15392-0.98-looksee.txt, 15392.wip.patch,
> 15392v2.wip.patch, 15392v3.wip.patch, HBASE-15392_suggest.patch,
> no_optimize.patch, no_optimize.patch, two_seeks.txt
>
>
> As found by Daniel "SystemTap" Pol, a simple Get results in our reading two
> HFileBlocks, the one that contains the wanted Cell, and the block that
> follows.
> Here is a bit of custom logging that logs a stack trace on each HFileBlock
> read so you can see the call stack responsible:
> {code}
> 2016-03-03 22:20:30,191 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] regionserver.StoreScanner:
> START LOOP
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] regionserver.StoreScanner:
> QCODE SEEK_NEXT_COL
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileBlockIndex:
> STARTED WHILE
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.CombinedBlockCache:
> OUT OF L2
> 2016-03-03 22:20:30,192 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] bucket.BucketCache: Read
> offset=31409152, len=2103
> 2016-03-03 22:20:30,192 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] bucket.FileIOEngine:
> offset=31409152, length=2103
> 2016-03-03 22:20:30,193 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileReaderImpl:
> From Cache [blockType=DATA, fileOffset=2055421, headerSize=33,
> onDiskSizeWithoutHeader=2024, uncompressedSizeWithoutHeader=2020,
> prevBlockOffset=2053364, isUseHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, onDiskDataSizeWithHeader=2053,
> getOnDiskSizeWithHeader=2057, totalChecksumBytes=4, isUnpacked=true,
> buf=[org.apache.hadoop.hbase.nio.SingleByteBuff@e19fbd54],
> dataBeginsWith=\x00\x00\x00)\x00\x00\x01`\x00\x16user995139035672819231,
> fileContext=[usesHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true,
> includesTags=false, compressAlgo=NONE, compressTags=false,
> cryptoContext=[cipher=NONE keyHash=NONE]]]
> 2016-03-03 22:20:30,193 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileReaderImpl:
> Cache hit return [blockType=DATA, fileOffset=2055421, headerSize=33,
> onDiskSizeWithoutHeader=2024, uncompressedSizeWithoutHeader=2020,
> prevBlockOffset=2053364, isUseHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, onDiskDataSizeWithHeader=2053,
> getOnDiskSizeWithHeader=2057, totalChecksumBytes=4, isUnpacked=true,
> buf=[org.apache.hadoop.hbase.nio.SingleByteBuff@e19fbd54],
> dataBeginsWith=\x00\x00\x00)\x00\x00\x01`\x00\x16user995139035672819231,
> fileContext=[usesHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true,
> includesTags=false, compressAlgo=NONE, compressTags=false,
> cryptoContext=[cipher=NONE keyHash=NONE]]]
> java.lang.Throwable
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1515)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$CellBasedKeyBlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:324)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:831)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.reseekTo(HFileReaderImpl.java:812)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:288)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:198)
> at
> org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:321)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java
[jira] [Updated] (HBASE-15377) Per-RS Get metric is time based, per-region metric is size-based
[
https://issues.apache.org/jira/browse/HBASE-15377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-15377:
--
Attachment: HBASE-15377.patch
With this patch, per-region has GetSize and GetTime metrics and RS has only
GetTime metrics.
> Per-RS Get metric is time based, per-region metric is size-based
>
>
> Key: HBASE-15377
> URL: https://issues.apache.org/jira/browse/HBASE-15377
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15377.patch
>
>
> We have metrics for Get operations at the region server level and region
> level.
> {code}
>"Get_num_ops" : 4837505,
> "Get_min" : 0,
> "Get_max" : 296,
> "Get_mean" : 0.2934618155433431,
> "Get_median" : 0.0,
> "Get_75th_percentile" : 0.0,
> "Get_95th_percentile" : 1.0,
> "Get_99th_percentile" : 1.0,
> {code}
> and
> {code}
>"Namespace_hbase_table_meta_region_1588230740_metric_get_num_ops" : 103,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_min" : 450,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_max" : 470,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_mean" :
> 450.19417475728153,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_median" : 460.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_75th_percentile"
> : 470.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_95th_percentile"
> : 470.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_99th_percentile"
> : 470.0,
> {code}
> The problem is that the report values for the region server shows the
> latency, versus the reported values for the region shows the response sizes.
> There is no way of telling this without reading the source code.
> I think we should deprecate response size histograms in favor of latency
> histograms.
> See also HBASE-15376.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15377) Per-RS Get metric is time based, per-region metric is size-based
[
https://issues.apache.org/jira/browse/HBASE-15377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-15377:
--
Assignee: Heng Chen
Fix Version/s: 1.4.0
1.3.0
2.0.0
Status: Patch Available (was: Reopened)
> Per-RS Get metric is time based, per-region metric is size-based
>
>
> Key: HBASE-15377
> URL: https://issues.apache.org/jira/browse/HBASE-15377
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Heng Chen
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15377.patch
>
>
> We have metrics for Get operations at the region server level and region
> level.
> {code}
>"Get_num_ops" : 4837505,
> "Get_min" : 0,
> "Get_max" : 296,
> "Get_mean" : 0.2934618155433431,
> "Get_median" : 0.0,
> "Get_75th_percentile" : 0.0,
> "Get_95th_percentile" : 1.0,
> "Get_99th_percentile" : 1.0,
> {code}
> and
> {code}
>"Namespace_hbase_table_meta_region_1588230740_metric_get_num_ops" : 103,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_min" : 450,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_max" : 470,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_mean" :
> 450.19417475728153,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_median" : 460.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_75th_percentile"
> : 470.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_95th_percentile"
> : 470.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_99th_percentile"
> : 470.0,
> {code}
> The problem is that the report values for the region server shows the
> latency, versus the reported values for the region shows the response sizes.
> There is no way of telling this without reading the source code.
> I think we should deprecate response size histograms in favor of latency
> histograms.
> See also HBASE-15376.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184369#comment-15184369 ] Enis Soztutar commented on HBASE-6721: -- bq. The longest component unit test run by a large margin is hbase-server and hbase-component I am not sure what is /component. Maybe [~busbey] can comment on it. At the worst we can run the tests locally (I can do that sometime this week). I saw a couple of checkstyle / findbugs issues. Did you have a chance to look into those? > RegionServer Group based Assignment > --- > > Key: HBASE-6721 > URL: https://issues.apache.org/jira/browse/HBASE-6721 > Project: HBase > Issue Type: New Feature >Reporter: Francis Liu >Assignee: Francis Liu > Labels: hbase-6721 > Attachments: 6721-master-webUI.patch, HBASE-6721 > GroupBasedLoadBalancer Sequence Diagram.xml, HBASE-6721-DesigDoc.pdf, > HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, > HBASE-6721_0.98_2.patch, HBASE-6721_10.patch, HBASE-6721_11.patch, > HBASE-6721_12.patch, HBASE-6721_13.patch, HBASE-6721_14.patch, > HBASE-6721_15.patch, HBASE-6721_8.patch, HBASE-6721_9.patch, > HBASE-6721_9.patch, HBASE-6721_94.patch, HBASE-6721_94.patch, > HBASE-6721_94_2.patch, HBASE-6721_94_3.patch, HBASE-6721_94_3.patch, > HBASE-6721_94_4.patch, HBASE-6721_94_5.patch, HBASE-6721_94_6.patch, > HBASE-6721_94_7.patch, HBASE-6721_98_1.patch, HBASE-6721_98_2.patch, > HBASE-6721_hbase-6721_addendum.patch, HBASE-6721_trunk.patch, > HBASE-6721_trunk.patch, HBASE-6721_trunk.patch, HBASE-6721_trunk1.patch, > HBASE-6721_trunk2.patch, balanceCluster Sequence Diagram.svg, > hbase-6721-v15-branch-1.1.patch, hbase-6721-v16.patch, hbase-6721-v17.patch, > hbase-6721-v18.patch, hbase-6721-v19.patch, hbase-6721-v20.patch, > hbase-6721-v21.patch, hbase-6721-v22.patch, hbase-6721-v23.patch, > hbase-6721-v25.patch, hbase-6721-v26.patch, hbase-6721-v26_draft1.patch, > hbase-6721-v27.patch, hbase-6721-v27.patch, hbase-6721-v27.patch.txt, > hbase-6721-v28.patch, hbase-6721-v28.patch, immediateAssignments Sequence > Diagram.svg, randomAssignment Sequence Diagram.svg, retainAssignment Sequence > Diagram.svg, roundRobinAssignment Sequence Diagram.svg > > > In multi-tenant deployments of HBase, it is likely that a RegionServer will > be serving out regions from a number of different tables owned by various > client applications. Being able to group a subset of running RegionServers > and assign specific tables to it, provides a client application a level of > isolation and resource allocation. > The proposal essentially is to have an AssignmentManager which is aware of > RegionServer groups and assigns tables to region servers based on groupings. > Load balancing will occur on a per group basis as well. > This is essentially a simplification of the approach taken in HBASE-4120. See > attached document. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15295) MutateTableAccess.multiMutate() does not get high priority causing a deadlock
[
https://issues.apache.org/jira/browse/HBASE-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184367#comment-15184367
]
Enis Soztutar commented on HBASE-15295:
---
bq. Thoughts on this one fellas? Would be good for 1.1.4.
Thanks Nick for taking a look. It is a bug fix, so we should have it in all
applicable I think. The patch is big, but somewhat mechanical mostly.
The bad thing is that if you are running with default number of handlers (30)
and sending 100 client threads to a phoenix table with secondary index, this is
easily reproducible.
I think the test failures are not related, but I'll look into it.
> MutateTableAccess.multiMutate() does not get high priority causing a deadlock
> -
>
> Key: HBASE-15295
> URL: https://issues.apache.org/jira/browse/HBASE-15295
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4
>
> Attachments: hbase-15295_v1.patch, hbase-15295_v1.patch,
> hbase-15295_v2.patch, hbase-15295_v3.patch, hbase-15295_v4.patch,
> hbase-15295_v5.patch, hbase-15295_v5.patch
>
>
> We have seen this in a cluster with Phoenix secondary indexes leading to a
> deadlock. All handlers are busy waiting on the index updates to finish:
> {code}
> "B.defaultRpcServer.handler=50,queue=0,port=16020" #91 daemon prio=5
> os_prio=0 tid=0x7f29f64ba000 nid=0xab51 waiting on condition
> [0x7f29a8762000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000124f1d5c8> (a
> com.google.common.util.concurrent.AbstractFuture$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:275)
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:66)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99)
> at
> org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter.write(ParallelWriterIndexCommitter.java:194)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:179)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:144)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:134)
> at
> org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:457)
> at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:406)
> at
> org.apache.phoenix.hbase.index.Indexer.postBatchMutate(Indexer.java:401)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$36.call(RegionCoprocessorHost.java:1006)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1748)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutate(RegionCoprocessorHost.java:1002)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3162)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2801)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2743)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(R
[jira] [Commented] (HBASE-15354) Use same criteria for clearing meta cache for all operations
[ https://issues.apache.org/jira/browse/HBASE-15354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184362#comment-15184362 ] Hudson commented on HBASE-15354: SUCCESS: Integrated in HBase-1.2-IT #456 (See [https://builds.apache.org/job/HBase-1.2-IT/456/]) HBASE-15354 Use same criteria for clearing meta cache for all operations (antonov: rev 0fc287ad9f3bb64f996be06eea1c5be2068454d5) * hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionServerCallable.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMetaCache.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionAdminServiceCallable.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java > Use same criteria for clearing meta cache for all operations > > > Key: HBASE-15354 > URL: https://issues.apache.org/jira/browse/HBASE-15354 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 1.2.1 > > Attachments: HBASE-15354-V0.patch, HBASE-15354-V1.patch, > HBASE-15354-V2.patch, HBASE-15354-V3.patch, HBASE-15354-V4.patch, > HBASE-15354-V5.patch, HBASE-15354-v5-addendum.patch > > > Currently we do not clear/update meta cache for some special exceptions if > the operation went through AsyncProcess#submit like HTable#put calls. But, we > clear meta cache without checking for these special exceptions in case of > other operations like gets, deletes etc because they directly go through the > RpcRetryingCaller#callWithRetries instead of the AsyncProcess. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15261) Make Throwable t in DaughterOpener volatile
[ https://issues.apache.org/jira/browse/HBASE-15261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184363#comment-15184363 ] Hudson commented on HBASE-15261: SUCCESS: Integrated in HBase-1.2-IT #456 (See [https://builds.apache.org/job/HBase-1.2-IT/456/]) HBASE-15261 Make Throwable t in DaughterOpener volatile (Huaxiang Sun) (stack: rev 3474a9e23c78e92942bfde2e51aab91ea4ecb84e) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransactionImpl.java > Make Throwable t in DaughterOpener volatile > --- > > Key: HBASE-15261 > URL: https://issues.apache.org/jira/browse/HBASE-15261 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.4.0 > > Attachments: HBASE-15261-001.patch, HBASE-15261-001.patch > > > In the region split process, daughter regions are opened in different > threads, Throwable t is set in these threads and it is checked in the calling > thread. Need to make it volatile so the checking will not miss any exceptions > from opening daughter regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15322) HBase 1.1.3 crashing
[ https://issues.apache.org/jira/browse/HBASE-15322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-15322: - Assignee: Anoop Sam John > HBase 1.1.3 crashing > > > Key: HBASE-15322 > URL: https://issues.apache.org/jira/browse/HBASE-15322 > Project: HBase > Issue Type: Bug > Components: hbase >Affects Versions: 1.0.0, 2.0.0, 0.98.7, 0.94.24 > Environment: OS: Ubuntu 14.04/Ubuntu 15.10 > JDK: OpenJDK8/OpenJDK9 >Reporter: Anant Sharma >Assignee: Anoop Sam John >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 0.98.18, 1.4.0 > > Attachments: BASE-15322.patch > > > HBase crashes in standalone mode with the following log: > __ > 2016-02-24 22:38:37,578 ERROR [main] master.HMasterCommandLine: Master exiting > java.lang.RuntimeException: Failed construction of Master: class > org.apache.hadoop.hbase.master.HMaster > at > org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2341) > at > org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:233) > at > org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126) > at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2355) > Caused by: java.lang.NoClassDefFoundError: Could not initialize class > org.apache.hadoop.hbase.util.Bytes$LexicographicalComparerHolder$UnsafeComparer > at org.apache.hadoop.hbase.util.Bytes.putInt(Bytes.java:899) > at > org.apache.hadoop.hbase.KeyValue.createByteArray(KeyValue.java:1082) > at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:652) > at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:580) > at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:483) > at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:370) > at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:267) > at org.apache.hadoop.hbase.HConstants.(HConstants.java:978) > at > org.apache.hadoop.hbase.HTableDescriptor.(HTableDescriptor.java:1488) > at > org.apache.hadoop.hbase.util.FSTableDescriptors.(FSTableDescriptors.java:124) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:570) > at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:365) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2336) > __ > The class is in the hbase-common.jar and its there in the classpath as can be > seen from the log: > _ > 2016-02-24 22:38:32,538 INFO [main] util.ServerCommandLine: > env:CLASSPATH=/home/hduser/hbase/hbase-1.1.3:/home/hduser/hbase/hbase-1.1.3/lib/activation-1.1.jar:/home/hduser/hbase/hbase-1.1.3/lib/aopalliance-1.0.jar:/home/hduser/hbase/hbase-1.1.3/lib/apacheds-i18n-2.0.0-M15.jar:/home/hduser/hbase/hbase-1.1.3/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hduser/hbase/hbase-1.1.3/lib/api-asn1-api-1.0.0-M20.jar:/home/hduser/hbase/hbase-1.1.3/lib/api-util-1.0.0-M20.jar:/home/hduser/hbase/hbase-1.1.3/lib/asm-3.1.jar:/home/hduser/hbase/hbase-1.1.3/lib/avro-1.7.4.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-beanutils-1.7.0.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-beanutils-core-1.8.0.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-cli-1.2.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-codec-1.9.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-collections-3.2.2.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-compress-1.4.1.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-configuration-1.6.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-daemon-1.0.13.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-digester-1.8.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-el-1.0.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-httpclient-3.1.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-io-2.4.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-lang-2.6.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-logging-1.2.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-math-2.2.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-math3-3.1.1.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-n
[jira] [Commented] (HBASE-13590) TestEnableTableHandler.testEnableTableWithNoRegionServers is flakey
[ https://issues.apache.org/jira/browse/HBASE-13590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184341#comment-15184341 ] Nick Dimiduk commented on HBASE-13590: -- What happened here [~stack], [~carp84]? I find this one on branch-1, branch-1.3, branch-1.2, branch-1.1 but not master. Is it not needed there? > TestEnableTableHandler.testEnableTableWithNoRegionServers is flakey > --- > > Key: HBASE-13590 > URL: https://issues.apache.org/jira/browse/HBASE-13590 > Project: HBase > Issue Type: Test > Components: master >Reporter: Nick Dimiduk >Assignee: Yu Li > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4 > > Attachments: HBASE-13590.branch-1.1.patch, > HBASE-13590.branch-1.1.patch, HBASE-13590.branch-1.1.patch, > HBASE-13590.branch-1.patch, HBASE-13590.branch-1.v2.patch, > testEnableTableHandler_branch-1.1.log.zip, > testEnableTableHandler_branch-1.log.zip > > > Looking at our [build > history|https://builds.apache.org/job/HBase-1.1/buildTimeTrend], it seems > this test is flakey. See builds 429, 431, 439. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15295) MutateTableAccess.multiMutate() does not get high priority causing a deadlock
[
https://issues.apache.org/jira/browse/HBASE-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184334#comment-15184334
]
Nick Dimiduk commented on HBASE-15295:
--
Thoughts on this one fellas? Would be good for 1.1.4.
> MutateTableAccess.multiMutate() does not get high priority causing a deadlock
> -
>
> Key: HBASE-15295
> URL: https://issues.apache.org/jira/browse/HBASE-15295
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4
>
> Attachments: hbase-15295_v1.patch, hbase-15295_v1.patch,
> hbase-15295_v2.patch, hbase-15295_v3.patch, hbase-15295_v4.patch,
> hbase-15295_v5.patch, hbase-15295_v5.patch
>
>
> We have seen this in a cluster with Phoenix secondary indexes leading to a
> deadlock. All handlers are busy waiting on the index updates to finish:
> {code}
> "B.defaultRpcServer.handler=50,queue=0,port=16020" #91 daemon prio=5
> os_prio=0 tid=0x7f29f64ba000 nid=0xab51 waiting on condition
> [0x7f29a8762000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000124f1d5c8> (a
> com.google.common.util.concurrent.AbstractFuture$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:275)
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:66)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99)
> at
> org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter.write(ParallelWriterIndexCommitter.java:194)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:179)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:144)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:134)
> at
> org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:457)
> at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:406)
> at
> org.apache.phoenix.hbase.index.Indexer.postBatchMutate(Indexer.java:401)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$36.call(RegionCoprocessorHost.java:1006)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1748)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutate(RegionCoprocessorHost.java:1002)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3162)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2801)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2743)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> And the index region is trying to split, and is trying to do a meta update:
> {code}
> "regionserver//10.132.70.191:16020-splits-1454693389669" #1779
> prio=5 os_prio=0 tid=0x7f29e149c000 nid=0x5107 in Object.wait()
> [0x7f1f136d6000]
[jira] [Updated] (HBASE-15168) Zombie stomping branch-1.1 edition
[ https://issues.apache.org/jira/browse/HBASE-15168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-15168: - Fix Version/s: (was: 1.1.4) 1.1.5 > Zombie stomping branch-1.1 edition > -- > > Key: HBASE-15168 > URL: https://issues.apache.org/jira/browse/HBASE-15168 > Project: HBase > Issue Type: Umbrella > Components: test >Affects Versions: 1.1.0 >Reporter: Nick Dimiduk >Priority: Critical > Fix For: 1.1.5 > > > Let's bring back the work done on HBASE-14420 for branch-1.1, stabilize our > [builds|https://builds.apache.org/job/HBase-1.1-JDK7/]. Hang tickets here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13627) Terminating RS results in redundant CLOSE RPC
[
https://issues.apache.org/jira/browse/HBASE-13627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-13627:
-
Fix Version/s: (was: 1.1.4)
1.1.5
> Terminating RS results in redundant CLOSE RPC
> -
>
> Key: HBASE-13627
> URL: https://issues.apache.org/jira/browse/HBASE-13627
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 1.1.0
>Reporter: Nick Dimiduk
>Assignee: Sreeram Venkatasubramanian
>Priority: Minor
> Labels: beginner, beginners
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.5
>
>
> Noticed while testing the 1.1.0RC0 bits. It seems we're issuing a redundant
> close RPC during shutdown. This results in a logging warning for each region.
> {noformat}
> 2015-05-06 00:07:19,214 INFO [RS:0;ndimiduk-apache-1-1-dist-6:56371]
> regionserver.HRegionServer: Received CLOSE for the region:
> 19cbe4fe2fe5335e7aace05e10e36ede, which we are already trying to CLOSE, but
> not completed yet
> 2015-05-06 00:07:19,214 WARN [RS:0;ndimiduk-apache-1-1-dist-6:56371]
> regionserver.HRegionServer: Failed to close
> cluster_test,,1430869443384.19cbe4fe2fe5335e7aace05e10e36ede. -
> ignoring and continuing
> org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The
> region 19cbe4fe2fe5335e7aace05e10e36ede was already closing. New CLOSE
> request is ignored.
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2769)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegionIgnoreErrors(HRegionServer.java:2695)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.closeUserRegions(HRegionServer.java:2327)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:937)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> 1. launch a standalone cluster from tgz (./bin/start-hbase.sh)
> 2. load some data (ie, run bin/hbase ltt)
> 3. terminate cluster (./bin/stop-hbase.sh)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13603) Write test asserting desired priority of RS->Master RPCs
[
https://issues.apache.org/jira/browse/HBASE-13603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-13603:
-
Fix Version/s: (was: 1.1.4)
1.1.5
> Write test asserting desired priority of RS->Master RPCs
>
>
> Key: HBASE-13603
> URL: https://issues.apache.org/jira/browse/HBASE-13603
> Project: HBase
> Issue Type: Test
> Components: IPC/RPC, test
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.5
>
>
> From HBASE-13351:
> {quote}
> Any way we can write a FT test to assert that the RS->Master APIs are treated
> with higher priority. I see your UT for asserting the annotation.
> {quote}
> Write a test that verifies expected RPCs are run on the correct pools in as
> real of an environment possible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-11819: - Fix Version/s: (was: 1.1.4) 1.1.5 > Unit test for CoprocessorHConnection > - > > Key: HBASE-11819 > URL: https://issues.apache.org/jira/browse/HBASE-11819 > Project: HBase > Issue Type: Test >Reporter: Andrew Purtell >Assignee: Talat UYARER >Priority: Minor > Labels: beginner > Fix For: 2.0.0, 0.98.14, 1.3.0, 1.2.1, 1.1.5 > > Attachments: HBASE-11819v4-master.patch, HBASE-11819v5-0.98 > (1).patch, HBASE-11819v5-0.98.patch, HBASE-11819v5-master (1).patch, > HBASE-11819v5-master.patch, HBASE-11819v5-master.patch, > HBASE-11819v5-v0.98.patch, HBASE-11819v5-v1.0.patch > > > Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15387) Make HalfStoreFileReader configurable in LoadIncrementalHFiles
[ https://issues.apache.org/jira/browse/HBASE-15387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-15387: - Fix Version/s: (was: 1.1.4) 1.1.5 > Make HalfStoreFileReader configurable in LoadIncrementalHFiles > -- > > Key: HBASE-15387 > URL: https://issues.apache.org/jira/browse/HBASE-15387 > Project: HBase > Issue Type: Improvement >Reporter: Rajeshbabu Chintaguntla >Assignee: Rajeshbabu Chintaguntla > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.5 > > > Currently we are initializing HalfStoreFileReader to split the HFile but we > might have different implementation for splitting. So we can make it > configurable. It's needed for local indexing in Phoenix(PHOENIX-2736). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13354) Add documentation and tests for external block cache.
[ https://issues.apache.org/jira/browse/HBASE-13354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-13354: - Fix Version/s: (was: 1.1.4) 1.1.5 > Add documentation and tests for external block cache. > - > > Key: HBASE-13354 > URL: https://issues.apache.org/jira/browse/HBASE-13354 > Project: HBase > Issue Type: Bug > Components: BlockCache, documentation, test >Affects Versions: 2.0.0, 1.1.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.5 > > > The new memcached integration needs some documentation and some testing > showing how it works and what can go wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13160) SplitLogWorker does not pick up the task immediately
[
https://issues.apache.org/jira/browse/HBASE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-13160:
-
Fix Version/s: (was: 1.1.4)
1.1.5
> SplitLogWorker does not pick up the task immediately
>
>
> Key: HBASE-13160
> URL: https://issues.apache.org/jira/browse/HBASE-13160
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.5
>
> Attachments: hbase-13160_v1.patch
>
>
> We were reading some code with Jeffrey, and we realized that the
> SplitLogWorker's internal task loop is weird. It does {{ls}} every second and
> sleeps, but have another mechanism to learn about new tasks, but does not
> make affective use of the zk notification.
> I have a simple patch which might improve this area.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13587) TestSnapshotCloneIndependence.testOnlineSnapshotDeleteIndependent is flakey
[ https://issues.apache.org/jira/browse/HBASE-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-13587: - Fix Version/s: (was: 1.1.4) 1.1.5 > TestSnapshotCloneIndependence.testOnlineSnapshotDeleteIndependent is flakey > --- > > Key: HBASE-13587 > URL: https://issues.apache.org/jira/browse/HBASE-13587 > Project: HBase > Issue Type: Test >Reporter: Nick Dimiduk > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.5 > > > Looking at our [build > history|https://builds.apache.org/job/HBase-1.1/buildTimeTrend], it seems > this test is flakey. See builds 428, 431, 432, 433. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14391) Empty regionserver WAL will never be deleted although the coresponding regionserver has been stale
[
https://issues.apache.org/jira/browse/HBASE-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-14391:
-
Fix Version/s: (was: 1.1.4)
1.1.5
> Empty regionserver WAL will never be deleted although the coresponding
> regionserver has been stale
> --
>
> Key: HBASE-14391
> URL: https://issues.apache.org/jira/browse/HBASE-14391
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.0.2
>Reporter: Qianxi Zhang
>Assignee: Qianxi Zhang
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.5
>
> Attachments: HBASE-14391-master-v3.patch,
> HBASE_14391_master_v4.patch, HBASE_14391_trunk_v1.patch,
> HBASE_14391_trunk_v2.patch, WALs-leftover-dir.txt
>
>
> When I restarted the hbase cluster in which there was few data, I found there
> are two directories for one host with different timestamp which indicates
> that the old regionserver wal directory is not deleted.
> FHLog#989
> {code}
> @Override
> public void close() throws IOException {
> shutdown();
> final FileStatus[] files = getFiles();
> if (null != files && 0 != files.length) {
> for (FileStatus file : files) {
> Path p = getWALArchivePath(this.fullPathArchiveDir, file.getPath());
> // Tell our listeners that a log is going to be archived.
> if (!this.listeners.isEmpty()) {
> for (WALActionsListener i : this.listeners) {
> i.preLogArchive(file.getPath(), p);
> }
> }
> if (!FSUtils.renameAndSetModifyTime(fs, file.getPath(), p)) {
> throw new IOException("Unable to rename " + file.getPath() + " to "
> + p);
> }
> // Tell our listeners that a log was archived.
> if (!this.listeners.isEmpty()) {
> for (WALActionsListener i : this.listeners) {
> i.postLogArchive(file.getPath(), p);
> }
> }
> }
> LOG.debug("Moved " + files.length + " WAL file(s) to " +
> FSUtils.getPath(this.fullPathArchiveDir));
> }
> LOG.info("Closed WAL: " + toString());
> }
> {code}
> When regionserver is stopped, the hlog will be archived, so wal/regionserver
> is empty in hdfs.
> MasterFileSystem#252
> {code}
> if (curLogFiles == null || curLogFiles.length == 0) {
> // Empty log folder. No recovery needed
> continue;
> }
> {code}
> The regionserver directory will be not splitted, it makes sense. But it will
> be not deleted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15033) Backport test-patch.sh and zombie-detector.sh from master to branch-1.0/1.1
[ https://issues.apache.org/jira/browse/HBASE-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-15033: - Fix Version/s: (was: 1.1.4) 1.1.5 > Backport test-patch.sh and zombie-detector.sh from master to branch-1.0/1.1 > --- > > Key: HBASE-15033 > URL: https://issues.apache.org/jira/browse/HBASE-15033 > Project: HBase > Issue Type: Task > Components: build >Reporter: stack >Assignee: stack > Fix For: 1.0.3, 1.1.5 > > Attachments: 15033.patch > > > Backport current test-patch.sh and zombie dettector to branch-1.0+ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15185) Fix jdk8 javadoc warnings for branch-1.1
[ https://issues.apache.org/jira/browse/HBASE-15185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-15185: - Fix Version/s: (was: 1.1.4) 1.1.5 > Fix jdk8 javadoc warnings for branch-1.1 > > > Key: HBASE-15185 > URL: https://issues.apache.org/jira/browse/HBASE-15185 > Project: HBase > Issue Type: Task >Affects Versions: 1.1.3 >Reporter: Yu Li >Assignee: Yu Li > Fix For: 1.1.5 > > Attachments: HBASE-15185.branch-1.1.patch, > HBASE-15185.branch-1.1.v2.patch, HBASE-15185.branch-1.1.v3.patch > > > [This > link|https://builds.apache.org/job/PreCommit-HBASE-Build/340/artifact/patchprocess/patch-javadoc-hbase-server-jdk1.8.0_66.txt] > shows jdk8 javadoc warnings for current branch-1.1 code base. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15308) Flakey TestSplitWalDataLoss on branch-1.1
[ https://issues.apache.org/jira/browse/HBASE-15308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-15308: - Fix Version/s: (was: 1.1.4) 1.1.5 > Flakey TestSplitWalDataLoss on branch-1.1 > - > > Key: HBASE-15308 > URL: https://issues.apache.org/jira/browse/HBASE-15308 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Heng Chen > Fix For: 1.1.5 > > > It happens during HBASE-15169 QA test, see > https://builds.apache.org/job/PreCommit-HBASE-Build/628/artifact/patchprocess/patch-unit-hbase-server-jdk1.8.0_72.txt > https://builds.apache.org/job/PreCommit-HBASE-Build/547/artifact/patchprocess/patch-unit-hbase-server-jdk1.8.0_72.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14610) IntegrationTestRpcClient from HBASE-14535 is failing with Async RPC client
[ https://issues.apache.org/jira/browse/HBASE-14610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-14610: - Fix Version/s: (was: 1.1.4) 1.1.5 > IntegrationTestRpcClient from HBASE-14535 is failing with Async RPC client > -- > > Key: HBASE-14610 > URL: https://issues.apache.org/jira/browse/HBASE-14610 > Project: HBase > Issue Type: Bug > Components: IPC/RPC >Reporter: Enis Soztutar > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.0.4, 1.1.5 > > Attachments: output > > > HBASE-14535 introduces an IT to simulate a running cluster with RPC servers > and RPC clients doing requests against the servers. > It passes with the sync client, but fails with async client. Probably we need > to take a look. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14223) Meta WALs are not cleared if meta region was closed and RS aborts
[
https://issues.apache.org/jira/browse/HBASE-14223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-14223:
-
Fix Version/s: (was: 1.1.4)
1.1.5
> Meta WALs are not cleared if meta region was closed and RS aborts
> -
>
> Key: HBASE-14223
> URL: https://issues.apache.org/jira/browse/HBASE-14223
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.0.4, 1.1.5
>
> Attachments: HBASE-14223logs, hbase-14223_v0.patch,
> hbase-14223_v1-branch-1.patch, hbase-14223_v2-branch-1.patch,
> hbase-14223_v3-branch-1.patch, hbase-14223_v3-branch-1.patch,
> hbase-14223_v3-master.patch
>
>
> When an RS opens meta, and later closes it, the WAL(FSHlog) is not closed.
> The last WAL file just sits there in the RS WAL directory. If RS stops
> gracefully, the WAL file for meta is deleted. Otherwise if RS aborts, WAL for
> meta is not cleaned. It is also not split (which is correct) since master
> determines that the RS no longer hosts meta at the time of RS abort.
> From a cluster after running ITBLL with CM, I see a lot of {{-splitting}}
> directories left uncleaned:
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls
> /apps/hbase/data/WALs
> Found 31 items
> drwxr-xr-x - hbase hadoop 0 2015-06-05 01:14
> /apps/hbase/data/WALs/hregion-58203265
> drwxr-xr-x - hbase hadoop 0 2015-06-05 07:54
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433489308745-splitting
> drwxr-xr-x - hbase hadoop 0 2015-06-05 09:28
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433494382959-splitting
> drwxr-xr-x - hbase hadoop 0 2015-06-05 10:01
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433498252205-splitting
> ...
> {code}
> The directories contain WALs from meta:
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting
> Found 2 items
> -rw-r--r-- 3 hbase hadoop 201608 2015-06-05 03:15
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
> -rw-r--r-- 3 hbase hadoop 44420 2015-06-05 04:36
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> The RS hosted the meta region for some time:
> {code}
> 2015-06-05 03:14:28,692 INFO [PostOpenDeployTasks:1588230740]
> zookeeper.MetaTableLocator: Setting hbase:meta region location in ZooKeeper
> as os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285
> ...
> 2015-06-05 03:15:17,302 INFO
> [RS_CLOSE_META-os-enis-dal-test-jun-4-5:16020-0] regionserver.HRegion: Closed
> hbase:meta,,1.1588230740
> {code}
> In between, a WAL is created:
> {code}
> 2015-06-05 03:15:11,707 INFO
> [RS_OPEN_META-os-enis-dal-test-jun-4-5:16020-0-MetaLogRoller] wal.FSHLog:
> Rolled WAL
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
> with entries=385, filesize=196.88 KB; new WAL
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> When CM killed the region server later master did not see these WAL files:
> {code}
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:46,075
> INFO [MASTER_SERVER_OPERATIONS-os-enis-dal-test-jun-4-3:16000-0]
> master.SplitLogManager: started splitting 2 logs in
> [hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting]
> for [os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285]
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:47,300
> INFO [main-EventThread] wal.WALSplitter: Archived processed log
> hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285.default.1433475074436
> to
> hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/oldWALs/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285.default.1433475074436
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:50,497
> INFO [main-EventThread] wal.WALSplit
[jira] [Commented] (HBASE-14844) backport jdk.tools exclusion to 1.0 and 1.1
[ https://issues.apache.org/jira/browse/HBASE-14844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184327#comment-15184327 ] Nick Dimiduk commented on HBASE-14844: -- You still up for getting a solution here [~busbey]? I haven't heard anyone clamoring for this one, so I think we can put it off another month if necessary. Same with HBASE-14845. > backport jdk.tools exclusion to 1.0 and 1.1 > --- > > Key: HBASE-14844 > URL: https://issues.apache.org/jira/browse/HBASE-14844 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 1.1.2, 1.0.3 >Reporter: Sean Busbey >Priority: Critical > Fix For: 1.1.4, 1.0.4 > > > per [~apurtell]'s comment when backporting HBASE-13963 to 0.98, we should > probably consider leaking jdk.tools in 1.0 and 1.1 bugs as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15322) HBase 1.1.3 crashing
[
https://issues.apache.org/jira/browse/HBASE-15322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184326#comment-15184326
]
Nick Dimiduk commented on HBASE-15322:
--
This patch fixes your runtime [~koko.an...@gmail.com]?
{{UnsafeAccess}} is still referring directly to {{Unsafe}} -- this is
acceptable so long as all invocations of {{UnsafeAccess}} are guarded by checks
to {{UnsafeAvailChecker}}?
nit: You have a static constant for "sun.misc.Unsafe" but not for
"java.nio.Bits"; do both or neither. The rest looks good to me, +1 for fix on
commit. [~ram_krish] you get your concerns addressed?
> HBase 1.1.3 crashing
>
>
> Key: HBASE-15322
> URL: https://issues.apache.org/jira/browse/HBASE-15322
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 1.0.0, 2.0.0, 0.98.7, 0.94.24
> Environment: OS: Ubuntu 14.04/Ubuntu 15.10
> JDK: OpenJDK8/OpenJDK9
>Reporter: Anant Sharma
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 0.98.18, 1.4.0
>
> Attachments: BASE-15322.patch
>
>
> HBase crashes in standalone mode with the following log:
> __
> 2016-02-24 22:38:37,578 ERROR [main] master.HMasterCommandLine: Master exiting
> java.lang.RuntimeException: Failed construction of Master: class
> org.apache.hadoop.hbase.master.HMaster
> at
> org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2341)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:233)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2355)
> Caused by: java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.hbase.util.Bytes$LexicographicalComparerHolder$UnsafeComparer
> at org.apache.hadoop.hbase.util.Bytes.putInt(Bytes.java:899)
> at
> org.apache.hadoop.hbase.KeyValue.createByteArray(KeyValue.java:1082)
> at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:652)
> at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:580)
> at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:483)
> at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:370)
> at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:267)
> at org.apache.hadoop.hbase.HConstants.(HConstants.java:978)
> at
> org.apache.hadoop.hbase.HTableDescriptor.(HTableDescriptor.java:1488)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.(FSTableDescriptors.java:124)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:570)
> at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:365)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2336)
> __
> The class is in the hbase-common.jar and its there in the classpath as can be
> seen from the log:
> _
> 2016-02-24 22:38:32,538 INFO [main] util.ServerCommandLine:
> env:CLASSPATH=/home/hduser/hbase/hbase-1.1.3:/home/hduser/hbase/hbase-1.1.3/lib/activation-1.1.jar:/home/hduser/hbase/hbase-1.1.3/lib/aopalliance-1.0.jar:/home/hduser/hbase/hbase-1.1.3/lib/apacheds-i18n-2.0.0-M15.jar:/home/hduser/hbase/hbase-1.1.3/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hduser/hbase/hbase-1.1.3/lib/api-asn1-api-1.0.0-M20.jar:/home/hduser/hbase/hbase-1.1.3/lib/api-util-1.0.0-M20.jar:/home/hduser/hbase/hbase-1.1.3/lib/asm-3.1.jar:/home/hduser/hbase/hbase-1.1.3/lib/avro-1.7.4.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-beanutils-1.7.0.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-beanutils-core-1.8.0.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-cli-1.2.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-codec-1.9.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-collections-3.2.2.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-compress-1.4.1.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-configuration-1.6.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-daemon-1.0.13.jar:/home/hduser/hbase/hbase-1.1.3/lib/commons-digester-1.8.jar:/home/hduser/hbas
[jira] [Updated] (HBASE-15243) Utilize the lowest seek value when all Filters in MUST_PASS_ONE FilterList return SEEK_NEXT_USING_HINT
[ https://issues.apache.org/jira/browse/HBASE-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15243: --- Resolution: Fixed Fix Version/s: (was: 1.3.0) 1.4.0 Status: Resolved (was: Patch Available) Thanks for the reviews, Preston, Stack, Lars and Anoop. > Utilize the lowest seek value when all Filters in MUST_PASS_ONE FilterList > return SEEK_NEXT_USING_HINT > -- > > Key: HBASE-15243 > URL: https://issues.apache.org/jira/browse/HBASE-15243 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15243-v1.txt, HBASE-15243-v2.txt, > HBASE-15243-v3.txt, HBASE-15243-v4.txt, HBASE-15243-v5.txt, > HBASE-15243-v6.txt, HBASE-15243-v7.txt > > > As Preston Koprivica pointed out at the tail of HBASE-4394, when all filters > in a MUST_PASS_ONE FilterList return a SEEK_USING_NEXT_HINT code, we should > return SEEK_NEXT_USING_HINT from the FilterList to utilize the lowest seek > value. > This would save unnecessary accesses to blocks which are not in scan result. > A unit test has been added which shows the reduction in the number of blocks > accessed when this optimization takes effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14703) HTable.mutateRow does not collect stats
[ https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184322#comment-15184322 ] Andrew Purtell commented on HBASE-14703: Yeah it gets confusing because we have RMs on minor 1.x branches now where in the pre 1.0 days the RM is in charge of a major line (98, 94, etc). So I make decisions where each 0.98.x release is a minor release, and RMs for 1.x branches make decisions where each release there will only be a patch release. We don't have a RM for branch-1. Maybe we should always have a RM in waiting for that. Food for thought. > HTable.mutateRow does not collect stats > --- > > Key: HBASE-14703 > URL: https://issues.apache.org/jira/browse/HBASE-14703 > Project: HBase > Issue Type: Bug >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 2.0.0 > > Attachments: HBASE-14702_v5.2_addendum-addendum.patch, > HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch, > HBASE-14703-start.patch, HBASE-14703-v4.1.patch, HBASE-14703-v4.patch, > HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch, > HBASE-14703_v1.patch, HBASE-14703_v10.patch, HBASE-14703_v10.patch, > HBASE-14703_v11.patch, HBASE-14703_v12.patch, HBASE-14703_v13.patch, > HBASE-14703_v2.patch, HBASE-14703_v3.patch, HBASE-14703_v5.1.patch, > HBASE-14703_v5.2.patch, HBASE-14703_v5.patch, HBASE-14703_v6-addendum.patch, > HBASE-14703_v6.patch, HBASE-14703_v7.patch, HBASE-14703_v8.patch, > HBASE-14703_v9.patch > > > We are trying to fix the stats implementation, by moving it out of the Result > object and into an Rpc payload (but not the 'cell payload', just as part of > the values returned from the request). This change will also us use easily > switch to AsyncProcess as the executor, and support stats, for nearly all the > rpc calls. However, that means when you upgrade the client or server, you > will lose stats visibility until the other side is upgraded. We could keep > around the Result based stats storage to accommodate the old api and send > both stats back from the server (in each result and in the rpc payload). > Note that we will still be wire compatible - protobufs mean we can just ride > over the lack of information. > The other tricky part of this is that Result has a > non-InterfaceAudience.Private getStatistics() method (along with two > InterfaceAudience.Private addResults and setStatistics methods), so we might > need a release to deprecate the getStats() method before throwing it out? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15420) TestCacheConfig failed after HBASE-15338
[ https://issues.apache.org/jira/browse/HBASE-15420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184319#comment-15184319 ] Duo Zhang commented on HBASE-15420: --- +1 > TestCacheConfig failed after HBASE-15338 > > > Key: HBASE-15420 > URL: https://issues.apache.org/jira/browse/HBASE-15420 > Project: HBase > Issue Type: Test > Components: test >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15420-v1.diff > > > TestCacheConfig failed after HBASE-15338. > Fix it in this issue~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15354) Use same criteria for clearing meta cache for all operations
[ https://issues.apache.org/jira/browse/HBASE-15354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184314#comment-15184314 ] Hudson commented on HBASE-15354: SUCCESS: Integrated in HBase-1.3-IT #542 (See [https://builds.apache.org/job/HBase-1.3-IT/542/]) HBASE-15354 Use same criteria for clearing meta cache for all operations (antonov: rev 43698b3fcb6b6759c5acae3b889e6b280c2205bd) * hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionAdminServiceCallable.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMetaCache.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionServerCallable.java > Use same criteria for clearing meta cache for all operations > > > Key: HBASE-15354 > URL: https://issues.apache.org/jira/browse/HBASE-15354 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 1.2.1 > > Attachments: HBASE-15354-V0.patch, HBASE-15354-V1.patch, > HBASE-15354-V2.patch, HBASE-15354-V3.patch, HBASE-15354-V4.patch, > HBASE-15354-V5.patch, HBASE-15354-v5-addendum.patch > > > Currently we do not clear/update meta cache for some special exceptions if > the operation went through AsyncProcess#submit like HTable#put calls. But, we > clear meta cache without checking for these special exceptions in case of > other operations like gets, deletes etc because they directly go through the > RpcRetryingCaller#callWithRetries instead of the AsyncProcess. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15261) Make Throwable t in DaughterOpener volatile
[ https://issues.apache.org/jira/browse/HBASE-15261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184315#comment-15184315 ] Hudson commented on HBASE-15261: SUCCESS: Integrated in HBase-1.3-IT #542 (See [https://builds.apache.org/job/HBase-1.3-IT/542/]) HBASE-15261 Make Throwable t in DaughterOpener volatile (Huaxiang Sun) (stack: rev e42065c308f53123b947c4d36f19a3261c269cd3) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransactionImpl.java > Make Throwable t in DaughterOpener volatile > --- > > Key: HBASE-15261 > URL: https://issues.apache.org/jira/browse/HBASE-15261 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.4.0 > > Attachments: HBASE-15261-001.patch, HBASE-15261-001.patch > > > In the region split process, daughter regions are opened in different > threads, Throwable t is set in these threads and it is checked in the calling > thread. Need to make it volatile so the checking will not miss any exceptions > from opening daughter regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14703) HTable.mutateRow does not collect stats
[ https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184309#comment-15184309 ] Jesse Yates commented on HBASE-14703: - Great, thanks for the info [~apurtell]. I'll keep it in mind. > HTable.mutateRow does not collect stats > --- > > Key: HBASE-14703 > URL: https://issues.apache.org/jira/browse/HBASE-14703 > Project: HBase > Issue Type: Bug >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 2.0.0 > > Attachments: HBASE-14702_v5.2_addendum-addendum.patch, > HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch, > HBASE-14703-start.patch, HBASE-14703-v4.1.patch, HBASE-14703-v4.patch, > HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch, > HBASE-14703_v1.patch, HBASE-14703_v10.patch, HBASE-14703_v10.patch, > HBASE-14703_v11.patch, HBASE-14703_v12.patch, HBASE-14703_v13.patch, > HBASE-14703_v2.patch, HBASE-14703_v3.patch, HBASE-14703_v5.1.patch, > HBASE-14703_v5.2.patch, HBASE-14703_v5.patch, HBASE-14703_v6-addendum.patch, > HBASE-14703_v6.patch, HBASE-14703_v7.patch, HBASE-14703_v8.patch, > HBASE-14703_v9.patch > > > We are trying to fix the stats implementation, by moving it out of the Result > object and into an Rpc payload (but not the 'cell payload', just as part of > the values returned from the request). This change will also us use easily > switch to AsyncProcess as the executor, and support stats, for nearly all the > rpc calls. However, that means when you upgrade the client or server, you > will lose stats visibility until the other side is upgraded. We could keep > around the Result based stats storage to accommodate the old api and send > both stats back from the server (in each result and in the rpc payload). > Note that we will still be wire compatible - protobufs mean we can just ride > over the lack of information. > The other tricky part of this is that Result has a > non-InterfaceAudience.Private getStatistics() method (along with two > InterfaceAudience.Private addResults and setStatistics methods), so we might > need a release to deprecate the getStats() method before throwing it out? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14703) HTable.mutateRow does not collect stats
[ https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184307#comment-15184307 ] Andrew Purtell commented on HBASE-14703: No that's not how it is. The upstream branch of 0.98 is branch-1. 1.2 is a release branch made off of branch-1. Decisions Sean makes for 1.2 does not affect 0.98. Decisions we all make for branch-1 would. If we are not going to accept improvements or bug fixes for this feature into branch-1 and therefor 0.98 I consider it an abandoned feature and will submit and commit unless objection a patch to remove it. > HTable.mutateRow does not collect stats > --- > > Key: HBASE-14703 > URL: https://issues.apache.org/jira/browse/HBASE-14703 > Project: HBase > Issue Type: Bug >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 2.0.0 > > Attachments: HBASE-14702_v5.2_addendum-addendum.patch, > HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch, > HBASE-14703-start.patch, HBASE-14703-v4.1.patch, HBASE-14703-v4.patch, > HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch, > HBASE-14703_v1.patch, HBASE-14703_v10.patch, HBASE-14703_v10.patch, > HBASE-14703_v11.patch, HBASE-14703_v12.patch, HBASE-14703_v13.patch, > HBASE-14703_v2.patch, HBASE-14703_v3.patch, HBASE-14703_v5.1.patch, > HBASE-14703_v5.2.patch, HBASE-14703_v5.patch, HBASE-14703_v6-addendum.patch, > HBASE-14703_v6.patch, HBASE-14703_v7.patch, HBASE-14703_v8.patch, > HBASE-14703_v9.patch > > > We are trying to fix the stats implementation, by moving it out of the Result > object and into an Rpc payload (but not the 'cell payload', just as part of > the values returned from the request). This change will also us use easily > switch to AsyncProcess as the executor, and support stats, for nearly all the > rpc calls. However, that means when you upgrade the client or server, you > will lose stats visibility until the other side is upgraded. We could keep > around the Result based stats storage to accommodate the old api and send > both stats back from the server (in each result and in the rpc payload). > Note that we will still be wire compatible - protobufs mean we can just ride > over the lack of information. > The other tricky part of this is that Result has a > non-InterfaceAudience.Private getStatistics() method (along with two > InterfaceAudience.Private addResults and setStatistics methods), so we might > need a release to deprecate the getStats() method before throwing it out? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14703) HTable.mutateRow does not collect stats
[ https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184306#comment-15184306 ] Jesse Yates commented on HBASE-14703: - And to be fair, there was a long, unresolved discussion on the dev list as to what _should_ be backported to branches. Some of the argument was in favor of bring more things back, some of the argument was in favor of just bringing back bug fixes and keeping the improvements (which can have their own bugs) off the 'stable' branch. Again, this didn't get resolved, but doesn't help clarify project policies on this sort of stuff. I'm just trying to help here, hence garnering discussion around where it should go, especially as it is a kinda-sorta breaking change. > HTable.mutateRow does not collect stats > --- > > Key: HBASE-14703 > URL: https://issues.apache.org/jira/browse/HBASE-14703 > Project: HBase > Issue Type: Bug >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 2.0.0 > > Attachments: HBASE-14702_v5.2_addendum-addendum.patch, > HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch, > HBASE-14703-start.patch, HBASE-14703-v4.1.patch, HBASE-14703-v4.patch, > HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch, > HBASE-14703_v1.patch, HBASE-14703_v10.patch, HBASE-14703_v10.patch, > HBASE-14703_v11.patch, HBASE-14703_v12.patch, HBASE-14703_v13.patch, > HBASE-14703_v2.patch, HBASE-14703_v3.patch, HBASE-14703_v5.1.patch, > HBASE-14703_v5.2.patch, HBASE-14703_v5.patch, HBASE-14703_v6-addendum.patch, > HBASE-14703_v6.patch, HBASE-14703_v7.patch, HBASE-14703_v8.patch, > HBASE-14703_v9.patch > > > We are trying to fix the stats implementation, by moving it out of the Result > object and into an Rpc payload (but not the 'cell payload', just as part of > the values returned from the request). This change will also us use easily > switch to AsyncProcess as the executor, and support stats, for nearly all the > rpc calls. However, that means when you upgrade the client or server, you > will lose stats visibility until the other side is upgraded. We could keep > around the Result based stats storage to accommodate the old api and send > both stats back from the server (in each result and in the rpc payload). > Note that we will still be wire compatible - protobufs mean we can just ride > over the lack of information. > The other tricky part of this is that Result has a > non-InterfaceAudience.Private getStatistics() method (along with two > InterfaceAudience.Private addResults and setStatistics methods), so we might > need a release to deprecate the getStats() method before throwing it out? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15120) Backport HBASE-14883 to branch-1.1
[ https://issues.apache.org/jira/browse/HBASE-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184303#comment-15184303 ] Nick Dimiduk commented on HBASE-15120: -- Where are we at here. Can this one make 1.1.4? > Backport HBASE-14883 to branch-1.1 > -- > > Key: HBASE-15120 > URL: https://issues.apache.org/jira/browse/HBASE-15120 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.2 >Reporter: Yu Li >Assignee: Yu Li >Priority: Minor > Fix For: 1.1.4 > > Attachments: HBASE-15120.branch-1.1.patch, > HBASE-15120.branch-1.1.patch, HBASE-15120.branch-1.1.patch > > > When checking branch-1.1 UT in HBASE-13590, found > TestSplitTransactionOnCluster#testFailedSplit will fail at a 12/50 chance. > The issue is fixed by HBASE-14883 but the change didn't go into branch-1.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14703) HTable.mutateRow does not collect stats
[ https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184301#comment-15184301 ] Jesse Yates commented on HBASE-14703: - My understanding was that, _as a project_ we were not backporting features past an upstream branch point. So, if an upstream RM didn't want it, then the downstream shouldn't get it. IIRC, that came up for the client push back the first time around (but I'm probably thinking of another JIRA). Seeing as none of the other upstream RMs had weighed in, I took Sean's comments as what was expected to go in... he still RM'ing 1.2 line, yes? My apologies that I'm not up on the latest guidelines, as was going on previous discussions I had been involved with. You are still getting your code Mr. Purtell :) > HTable.mutateRow does not collect stats > --- > > Key: HBASE-14703 > URL: https://issues.apache.org/jira/browse/HBASE-14703 > Project: HBase > Issue Type: Bug >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 2.0.0 > > Attachments: HBASE-14702_v5.2_addendum-addendum.patch, > HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch, > HBASE-14703-start.patch, HBASE-14703-v4.1.patch, HBASE-14703-v4.patch, > HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch, > HBASE-14703_v1.patch, HBASE-14703_v10.patch, HBASE-14703_v10.patch, > HBASE-14703_v11.patch, HBASE-14703_v12.patch, HBASE-14703_v13.patch, > HBASE-14703_v2.patch, HBASE-14703_v3.patch, HBASE-14703_v5.1.patch, > HBASE-14703_v5.2.patch, HBASE-14703_v5.patch, HBASE-14703_v6-addendum.patch, > HBASE-14703_v6.patch, HBASE-14703_v7.patch, HBASE-14703_v8.patch, > HBASE-14703_v9.patch > > > We are trying to fix the stats implementation, by moving it out of the Result > object and into an Rpc payload (but not the 'cell payload', just as part of > the values returned from the request). This change will also us use easily > switch to AsyncProcess as the executor, and support stats, for nearly all the > rpc calls. However, that means when you upgrade the client or server, you > will lose stats visibility until the other side is upgraded. We could keep > around the Result based stats storage to accommodate the old api and send > both stats back from the server (in each result and in the rpc payload). > Note that we will still be wire compatible - protobufs mean we can just ride > over the lack of information. > The other tricky part of this is that Result has a > non-InterfaceAudience.Private getStatistics() method (along with two > InterfaceAudience.Private addResults and setStatistics methods), so we might > need a release to deprecate the getStats() method before throwing it out? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14703) HTable.mutateRow does not collect stats
[ https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184295#comment-15184295 ] Andrew Purtell commented on HBASE-14703: Also, 0.98 and branch-1 commits are not "gated" by arbitrary decisions of RMs not yet even nominated. We already have this feature in 0.98 and branch-1. Either we continue to accept improvements and bugfixes for it or we rip it out. > HTable.mutateRow does not collect stats > --- > > Key: HBASE-14703 > URL: https://issues.apache.org/jira/browse/HBASE-14703 > Project: HBase > Issue Type: Bug >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 2.0.0 > > Attachments: HBASE-14702_v5.2_addendum-addendum.patch, > HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch, > HBASE-14703-start.patch, HBASE-14703-v4.1.patch, HBASE-14703-v4.patch, > HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch, > HBASE-14703_v1.patch, HBASE-14703_v10.patch, HBASE-14703_v10.patch, > HBASE-14703_v11.patch, HBASE-14703_v12.patch, HBASE-14703_v13.patch, > HBASE-14703_v2.patch, HBASE-14703_v3.patch, HBASE-14703_v5.1.patch, > HBASE-14703_v5.2.patch, HBASE-14703_v5.patch, HBASE-14703_v6-addendum.patch, > HBASE-14703_v6.patch, HBASE-14703_v7.patch, HBASE-14703_v8.patch, > HBASE-14703_v9.patch > > > We are trying to fix the stats implementation, by moving it out of the Result > object and into an Rpc payload (but not the 'cell payload', just as part of > the values returned from the request). This change will also us use easily > switch to AsyncProcess as the executor, and support stats, for nearly all the > rpc calls. However, that means when you upgrade the client or server, you > will lose stats visibility until the other side is upgraded. We could keep > around the Result based stats storage to accommodate the old api and send > both stats back from the server (in each result and in the rpc payload). > Note that we will still be wire compatible - protobufs mean we can just ride > over the lack of information. > The other tricky part of this is that Result has a > non-InterfaceAudience.Private getStatistics() method (along with two > InterfaceAudience.Private addResults and setStatistics methods), so we might > need a release to deprecate the getStats() method before throwing it out? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14703) HTable.mutateRow does not collect stats
[ https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184291#comment-15184291 ] Andrew Purtell commented on HBASE-14703: With all due respect, Sean is not the RM for 0.98 > HTable.mutateRow does not collect stats > --- > > Key: HBASE-14703 > URL: https://issues.apache.org/jira/browse/HBASE-14703 > Project: HBase > Issue Type: Bug >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 2.0.0 > > Attachments: HBASE-14702_v5.2_addendum-addendum.patch, > HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch, > HBASE-14703-start.patch, HBASE-14703-v4.1.patch, HBASE-14703-v4.patch, > HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch, > HBASE-14703_v1.patch, HBASE-14703_v10.patch, HBASE-14703_v10.patch, > HBASE-14703_v11.patch, HBASE-14703_v12.patch, HBASE-14703_v13.patch, > HBASE-14703_v2.patch, HBASE-14703_v3.patch, HBASE-14703_v5.1.patch, > HBASE-14703_v5.2.patch, HBASE-14703_v5.patch, HBASE-14703_v6-addendum.patch, > HBASE-14703_v6.patch, HBASE-14703_v7.patch, HBASE-14703_v8.patch, > HBASE-14703_v9.patch > > > We are trying to fix the stats implementation, by moving it out of the Result > object and into an Rpc payload (but not the 'cell payload', just as part of > the values returned from the request). This change will also us use easily > switch to AsyncProcess as the executor, and support stats, for nearly all the > rpc calls. However, that means when you upgrade the client or server, you > will lose stats visibility until the other side is upgraded. We could keep > around the Result based stats storage to accommodate the old api and send > both stats back from the server (in each result and in the rpc payload). > Note that we will still be wire compatible - protobufs mean we can just ride > over the lack of information. > The other tricky part of this is that Result has a > non-InterfaceAudience.Private getStatistics() method (along with two > InterfaceAudience.Private addResults and setStatistics methods), so we might > need a release to deprecate the getStats() method before throwing it out? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15243) Utilize the lowest seek value when all Filters in MUST_PASS_ONE FilterList return SEEK_NEXT_USING_HINT
[ https://issues.apache.org/jira/browse/HBASE-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184289#comment-15184289 ] Hudson commented on HBASE-15243: FAILURE: Integrated in HBase-Trunk_matrix #762 (See [https://builds.apache.org/job/HBase-Trunk_matrix/762/]) HBASE-15243 Utilize the lowest seek value when all Filters in (tedyu: rev 05c1309b3c4ef5eb2634a84339f608fc4e694318) * hbase-client/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java * hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilterListOrOperatorWithBlkCnt.java HBASE-15243 Revert due to compilation error - master branch change makes (tedyu: rev dfc650a953c3a43e28a72736cd397c9bd41e787c) * hbase-client/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java * hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilterListOrOperatorWithBlkCnt.java HBASE-15243 Utilize the lowest seek value when all Filters in (tedyu: rev ed977fd125e4180559129739b198b158d5e84b48) * hbase-client/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java * hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilterListOrOperatorWithBlkCnt.java > Utilize the lowest seek value when all Filters in MUST_PASS_ONE FilterList > return SEEK_NEXT_USING_HINT > -- > > Key: HBASE-15243 > URL: https://issues.apache.org/jira/browse/HBASE-15243 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15243-v1.txt, HBASE-15243-v2.txt, > HBASE-15243-v3.txt, HBASE-15243-v4.txt, HBASE-15243-v5.txt, > HBASE-15243-v6.txt, HBASE-15243-v7.txt > > > As Preston Koprivica pointed out at the tail of HBASE-4394, when all filters > in a MUST_PASS_ONE FilterList return a SEEK_USING_NEXT_HINT code, we should > return SEEK_NEXT_USING_HINT from the FilterList to utilize the lowest seek > value. > This would save unnecessary accesses to blocks which are not in scan result. > A unit test has been added which shows the reduction in the number of blocks > accessed when this optimization takes effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15420) TestCacheConfig failed after HBASE-15338
[ https://issues.apache.org/jira/browse/HBASE-15420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184286#comment-15184286 ] Liu Shaohui commented on HBASE-15420: - [~Apache9] [~stack] [~anoop.hbase] Please help to review this patch? > TestCacheConfig failed after HBASE-15338 > > > Key: HBASE-15420 > URL: https://issues.apache.org/jira/browse/HBASE-15420 > Project: HBase > Issue Type: Test > Components: test >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15420-v1.diff > > > TestCacheConfig failed after HBASE-15338. > Fix it in this issue~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15420) TestCacheConfig failed after HBASE-15338
[ https://issues.apache.org/jira/browse/HBASE-15420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184282#comment-15184282 ] Liu Shaohui commented on HBASE-15420: - The test failed for patch v7 in HBASE-15338. In this version, the meta blocks don't need to be cached if cache on read is disabled. So just change the assertTrue to assertFalse to fix this failed tests~ > TestCacheConfig failed after HBASE-15338 > > > Key: HBASE-15420 > URL: https://issues.apache.org/jira/browse/HBASE-15420 > Project: HBase > Issue Type: Test > Components: test >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15420-v1.diff > > > TestCacheConfig failed after HBASE-15338. > Fix it in this issue~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15420) TestCacheConfig failed after HBASE-15338
[ https://issues.apache.org/jira/browse/HBASE-15420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-15420: Attachment: HBASE-15420-v1.diff Fix the failed test~ > TestCacheConfig failed after HBASE-15338 > > > Key: HBASE-15420 > URL: https://issues.apache.org/jira/browse/HBASE-15420 > Project: HBase > Issue Type: Test > Components: test >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15420-v1.diff > > > TestCacheConfig failed after HBASE-15338. > Fix it in this issue~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15420) TestCacheConfig failed after HBASE-15338
[ https://issues.apache.org/jira/browse/HBASE-15420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-15420: Status: Patch Available (was: Open) > TestCacheConfig failed after HBASE-15338 > > > Key: HBASE-15420 > URL: https://issues.apache.org/jira/browse/HBASE-15420 > Project: HBase > Issue Type: Test > Components: test >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15420-v1.diff > > > TestCacheConfig failed after HBASE-15338. > Fix it in this issue~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15420) TestCacheConfig failed after HBASE-15338
Liu Shaohui created HBASE-15420: --- Summary: TestCacheConfig failed after HBASE-15338 Key: HBASE-15420 URL: https://issues.apache.org/jira/browse/HBASE-15420 Project: HBase Issue Type: Test Components: test Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 2.0.0 TestCacheConfig failed after HBASE-15338. Fix it in this issue~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15377) Per-RS Get metric is time based, per-region metric is size-based
[
https://issues.apache.org/jira/browse/HBASE-15377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184278#comment-15184278
]
Heng Chen commented on HBASE-15377:
---
According to HBASE-15376, i think we could replace response size histograms
with latency histograms directly on branch-1.3+.wdyt?
> Per-RS Get metric is time based, per-region metric is size-based
>
>
> Key: HBASE-15377
> URL: https://issues.apache.org/jira/browse/HBASE-15377
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>
> We have metrics for Get operations at the region server level and region
> level.
> {code}
>"Get_num_ops" : 4837505,
> "Get_min" : 0,
> "Get_max" : 296,
> "Get_mean" : 0.2934618155433431,
> "Get_median" : 0.0,
> "Get_75th_percentile" : 0.0,
> "Get_95th_percentile" : 1.0,
> "Get_99th_percentile" : 1.0,
> {code}
> and
> {code}
>"Namespace_hbase_table_meta_region_1588230740_metric_get_num_ops" : 103,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_min" : 450,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_max" : 470,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_mean" :
> 450.19417475728153,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_median" : 460.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_75th_percentile"
> : 470.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_95th_percentile"
> : 470.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_99th_percentile"
> : 470.0,
> {code}
> The problem is that the report values for the region server shows the
> latency, versus the reported values for the region shows the response sizes.
> There is no way of telling this without reading the source code.
> I think we should deprecate response size histograms in favor of latency
> histograms.
> See also HBASE-15376.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15398) Cells loss or disorder when using family essential filter and partial scanning protocol
[ https://issues.apache.org/jira/browse/HBASE-15398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184273#comment-15184273 ] Ted Yu commented on HBASE-15398: bq. should the exception be thrown in server or client? I would say that if exception is needed on server side, we'd better reject the combination on client side. bq. it is a little complex after the patch I think so too. The code with patch is hard to understand / maintain. > Cells loss or disorder when using family essential filter and partial > scanning protocol > --- > > Key: HBASE-15398 > URL: https://issues.apache.org/jira/browse/HBASE-15398 > Project: HBase > Issue Type: Bug > Components: dataloss, Scanners >Affects Versions: 1.2.0, 1.1.3 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Critical > Attachments: 15398-test.txt, HBASE-15398.v1.txt > > > In RegionScannerImpl, we have two heaps, storeHeap and joinedHeap. If we have > a filter and it doesn't apply to all cf, the stores whose families needn't be > filtered will be in joinedHeap. We scan storeHeap first, then joinedHeap, > and merge the results and sort and return to client. We need sort because the > order of Cell is rowkey/cf/cq/ts and a smaller cf may be in the joinedHeap. > However, after HBASE-11544 we may transfer partial results when we get > SIZE_LIMIT_REACHED_MID_ROW or other similar states. We may return a larger cf > first because it is in storeHeap and then a smaller cf because it is in > joinedHeap. Server won't hold all cells in a row and client doesn't have a > sorting logic. The order of cf in Result for user is wrong. > And a more critical bug is, if we get a LIMIT_REACHED_MID_ROW on the last > cell of a row in storeHeap, we will break scanning in RegionScannerImpl and > in populateResult we will change the state to SIZE_LIMIT_REACHED because next > peeked cell is next row. But this is only the last cell of one and we have > two... And SIZE_LIMIT_REACHED means this Result is not partial (by > ScannerContext.partialResultFormed), client will see it and merge them and > return to user with losing data of joinedHeap. On next scan we will read next > row of storeHeap and joinedHeap is forgotten and never be read... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15338) Add a option to disable the data block cache for testing the performance of underlying file system
[ https://issues.apache.org/jira/browse/HBASE-15338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184271#comment-15184271 ] Liu Shaohui commented on HBASE-15338: - Sorry for that. I open a issue and fix it. > Add a option to disable the data block cache for testing the performance of > underlying file system > -- > > Key: HBASE-15338 > URL: https://issues.apache.org/jira/browse/HBASE-15338 > Project: HBase > Issue Type: Improvement > Components: integration tests >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15338-trunk-v1.diff, HBASE-15338-trunk-v2.diff, > HBASE-15338-trunk-v3.diff, HBASE-15338-trunk-v4.diff, > HBASE-15338-trunk-v5.diff, HBASE-15338-trunk-v6.diff, > HBASE-15338-trunk-v7.diff > > > When testing and comparing the performance of different file systems(HDFS, > Azure blob storage, AWS S3 and so on) for HBase, it's better to avoid the > affect of the HBase BlockCache and get the actually random read latency when > data block is read from underlying file system. (Usually, the index block and > meta block should be cached in memory in the testing). > So we add a option in CacheConfig to disable the data block cache. > Suggestions are welcomed~ Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14703) HTable.mutateRow does not collect stats
[ https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184268#comment-15184268 ] Heng Chen commented on HBASE-14703: --- Sounds good. I will give a patch for branch-1 this week. :) > HTable.mutateRow does not collect stats > --- > > Key: HBASE-14703 > URL: https://issues.apache.org/jira/browse/HBASE-14703 > Project: HBase > Issue Type: Bug >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 2.0.0 > > Attachments: HBASE-14702_v5.2_addendum-addendum.patch, > HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch, > HBASE-14703-start.patch, HBASE-14703-v4.1.patch, HBASE-14703-v4.patch, > HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch, > HBASE-14703_v1.patch, HBASE-14703_v10.patch, HBASE-14703_v10.patch, > HBASE-14703_v11.patch, HBASE-14703_v12.patch, HBASE-14703_v13.patch, > HBASE-14703_v2.patch, HBASE-14703_v3.patch, HBASE-14703_v5.1.patch, > HBASE-14703_v5.2.patch, HBASE-14703_v5.patch, HBASE-14703_v6-addendum.patch, > HBASE-14703_v6.patch, HBASE-14703_v7.patch, HBASE-14703_v8.patch, > HBASE-14703_v9.patch > > > We are trying to fix the stats implementation, by moving it out of the Result > object and into an Rpc payload (but not the 'cell payload', just as part of > the values returned from the request). This change will also us use easily > switch to AsyncProcess as the executor, and support stats, for nearly all the > rpc calls. However, that means when you upgrade the client or server, you > will lose stats visibility until the other side is upgraded. We could keep > around the Result based stats storage to accommodate the old api and send > both stats back from the server (in each result and in the rpc payload). > Note that we will still be wire compatible - protobufs mean we can just ride > over the lack of information. > The other tricky part of this is that Result has a > non-InterfaceAudience.Private getStatistics() method (along with two > InterfaceAudience.Private addResults and setStatistics methods), so we might > need a release to deprecate the getStats() method before throwing it out? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15377) Per-RS Get metric is time based, per-region metric is size-based
[
https://issues.apache.org/jira/browse/HBASE-15377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184264#comment-15184264
]
Heng Chen commented on HBASE-15377:
---
Oh, yeah, sorry for my careless. Let me give a patch later.
> Per-RS Get metric is time based, per-region metric is size-based
>
>
> Key: HBASE-15377
> URL: https://issues.apache.org/jira/browse/HBASE-15377
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>
> We have metrics for Get operations at the region server level and region
> level.
> {code}
>"Get_num_ops" : 4837505,
> "Get_min" : 0,
> "Get_max" : 296,
> "Get_mean" : 0.2934618155433431,
> "Get_median" : 0.0,
> "Get_75th_percentile" : 0.0,
> "Get_95th_percentile" : 1.0,
> "Get_99th_percentile" : 1.0,
> {code}
> and
> {code}
>"Namespace_hbase_table_meta_region_1588230740_metric_get_num_ops" : 103,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_min" : 450,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_max" : 470,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_mean" :
> 450.19417475728153,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_median" : 460.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_75th_percentile"
> : 470.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_95th_percentile"
> : 470.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_99th_percentile"
> : 470.0,
> {code}
> The problem is that the report values for the region server shows the
> latency, versus the reported values for the region shows the response sizes.
> There is no way of telling this without reading the source code.
> I think we should deprecate response size histograms in favor of latency
> histograms.
> See also HBASE-15376.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184252#comment-15184252
]
Heng Chen commented on HBASE-15406:
---
{quote}
I was thinking of using Leases to keep track of the running operation (like
HBCK) and if we tie a state to it (like disabling splits) then either the
client will come back and remove the lease + do cleanup, or the lease will
expire and the master will do the cleanup.
{quote}
Sounds good to me. We should extract this logic as running operation (not just
hbck). I really don't like one thread to do cleanup in master just for hbck.
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15413) Procedure-V2: print out ProcedureInfo during trace
[
https://issues.apache.org/jira/browse/HBASE-15413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184248#comment-15184248
]
Hadoop QA commented on HBASE-15413:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 48s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 1s
{color} | {color:green} hbase-common in the patch passed with JDK v1.8.0.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 5s
{color} | {color:green} hbase-common in the patch passed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 7s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12791864/HBASE-15413.v1-master.patch
|
| JIRA Issue | HBASE-15413 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-person
[jira] [Commented] (HBASE-15398) Cells loss or disorder when using family essential filter and partial scanning protocol
[ https://issues.apache.org/jira/browse/HBASE-15398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184234#comment-15184234 ] Phil Yang commented on HBASE-15398: --- Some tests failed, trying to fix them. And maybe I need a refactor on RegionSannerImpl.nextInternal , it is a little complex after the patch > Cells loss or disorder when using family essential filter and partial > scanning protocol > --- > > Key: HBASE-15398 > URL: https://issues.apache.org/jira/browse/HBASE-15398 > Project: HBase > Issue Type: Bug > Components: dataloss, Scanners >Affects Versions: 1.2.0, 1.1.3 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Critical > Attachments: 15398-test.txt, HBASE-15398.v1.txt > > > In RegionScannerImpl, we have two heaps, storeHeap and joinedHeap. If we have > a filter and it doesn't apply to all cf, the stores whose families needn't be > filtered will be in joinedHeap. We scan storeHeap first, then joinedHeap, > and merge the results and sort and return to client. We need sort because the > order of Cell is rowkey/cf/cq/ts and a smaller cf may be in the joinedHeap. > However, after HBASE-11544 we may transfer partial results when we get > SIZE_LIMIT_REACHED_MID_ROW or other similar states. We may return a larger cf > first because it is in storeHeap and then a smaller cf because it is in > joinedHeap. Server won't hold all cells in a row and client doesn't have a > sorting logic. The order of cf in Result for user is wrong. > And a more critical bug is, if we get a LIMIT_REACHED_MID_ROW on the last > cell of a row in storeHeap, we will break scanning in RegionScannerImpl and > in populateResult we will change the state to SIZE_LIMIT_REACHED because next > peeked cell is next row. But this is only the last cell of one and we have > two... And SIZE_LIMIT_REACHED means this Result is not partial (by > ScannerContext.partialResultFormed), client will see it and merge them and > return to user with losing data of joinedHeap. On next scan we will read next > row of storeHeap and joinedHeap is forgotten and never be read... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15338) Add a option to disable the data block cache for testing the performance of underlying file system
[
https://issues.apache.org/jira/browse/HBASE-15338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184230#comment-15184230
]
Duo Zhang commented on HBASE-15338:
---
https://builds.apache.org/job/HBase-Trunk_matrix/761/jdk=latest1.7,label=yahoo-not-h2/testReport/junit/org.apache.hadoop.hbase.io.hfile/TestCacheConfig/testDisableCacheDataBlock/
I tried locally, the testcase was also failed.
{noformat}
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.hbase.io.hfile.TestCacheConfig.testDisableCacheDataBlock(TestCacheConfig.java:258)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
{noformat}
But I can not reopen the issue because it is marked as closed...
[~liushaohui] Could you please open a new issue and solve this problem? Thanks.
> Add a option to disable the data block cache for testing the performance of
> underlying file system
> --
>
> Key: HBASE-15338
> URL: https://issues.apache.org/jira/browse/HBASE-15338
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-15338-trunk-v1.diff, HBASE-15338-trunk-v2.diff,
> HBASE-15338-trunk-v3.diff, HBASE-15338-trunk-v4.diff,
> HBASE-15338-trunk-v5.diff, HBASE-15338-trunk-v6.diff,
> HBASE-15338-trunk-v7.diff
>
>
> When testing and comparing the performance of different file systems(HDFS,
> Azure blob storage, AWS S3 and so on) for HBase, it's better to avoid the
> affect of the HBase BlockCache and get the actually random read latency when
> data block is read from underlying file system. (Usually, the index block and
> meta block should be cached in memory in the testing).
> So we add a option in CacheConfig to disable the data block cache.
> Suggestions are welcomed~ Thanks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15354) Use same criteria for clearing meta cache for all operations
[ https://issues.apache.org/jira/browse/HBASE-15354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-15354: Resolution: Fixed Fix Version/s: 1.2.1 Release Note: This patch fixes some issues when MetaCache (region location cache) gets unnecessarily dropped on the client. On master branch we now in RegionServerCallable and RegionServerAdminCallable pass the actual exception down to Connection#updateCachedLocation, so we could check there if the exception is "meta-clearing" or not. on branch-1, branch-1.2 and branch 1.3 we now check if the exception is meta-clearing or not in AsyncProcess (this check was there on master, but not on earlier branches) Status: Resolved (was: Patch Available) > Use same criteria for clearing meta cache for all operations > > > Key: HBASE-15354 > URL: https://issues.apache.org/jira/browse/HBASE-15354 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 1.2.1 > > Attachments: HBASE-15354-V0.patch, HBASE-15354-V1.patch, > HBASE-15354-V2.patch, HBASE-15354-V3.patch, HBASE-15354-V4.patch, > HBASE-15354-V5.patch, HBASE-15354-v5-addendum.patch > > > Currently we do not clear/update meta cache for some special exceptions if > the operation went through AsyncProcess#submit like HTable#put calls. But, we > clear meta cache without checking for these special exceptions in case of > other operations like gets, deletes etc because they directly go through the > RpcRetryingCaller#callWithRetries instead of the AsyncProcess. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15396) Enhance mapreduce.TableSplit to add encoded region name
[
https://issues.apache.org/jira/browse/HBASE-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184227#comment-15184227
]
Hadoop QA commented on HBASE-15396:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m 41s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s
{color} | {color:green} master passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} master passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
39s {color} | {color:green} hbase-server: patch generated 0 new + 23 unchanged
- 9 fixed = 23 total (was 32) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
39m 2s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 105m 45s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 191m 31s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.balancer.TestStochasticLoadBalancer2
|
| | hadoop.hbase.io.hfile.TestCacheConfig |
| Timed out junit tests |
org.apache.hadoop.hbase.regionserver.TestCorruptedRegionStoreFile |
| | org.apache.hadoop.hbase.regionserver.TestSplitLogWorker |
| | org.apache.hadoop.hbase.regionserver.compactions.TestFIFOCompactionPolicy
|
| | org.apache.hadoop.hbase.regionserver.TestCompaction |
| | org.apache.hadoop.hbase.regionserver.wal.TestFSHLog |
| | org.apache.hadoop.hbase.coprocessor.TestRegionObserverScannerOpenHook |
| | org.apache.hadoop.hbase.coprocessor.TestRegionServerObserver |
| | or
[jira] [Commented] (HBASE-15261) Make Throwable t in DaughterOpener volatile
[ https://issues.apache.org/jira/browse/HBASE-15261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184225#comment-15184225 ] Hiroshi Ikeda commented on HBASE-15261: --- Is there any possibility to refer the exception before joining or checking the termination of the thread? Otherwise that just wastes. > Make Throwable t in DaughterOpener volatile > --- > > Key: HBASE-15261 > URL: https://issues.apache.org/jira/browse/HBASE-15261 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.4.0 > > Attachments: HBASE-15261-001.patch, HBASE-15261-001.patch > > > In the region split process, daughter regions are opened in different > threads, Throwable t is set in these threads and it is checked in the calling > thread. Need to make it volatile so the checking will not miss any exceptions > from opening daughter regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15354) Use same criteria for clearing meta cache for all operations
[ https://issues.apache.org/jira/browse/HBASE-15354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-15354: Attachment: HBASE-15354-v5-addendum.patch Thanks [~busbey]. Committed to branch-1, branch-1.3, branch-1.2. There was an minicluster related issue which cause the new test to fail on branch-1, so I added this addendum (committed separately to master and included in cherry-pick to other branches) > Use same criteria for clearing meta cache for all operations > > > Key: HBASE-15354 > URL: https://issues.apache.org/jira/browse/HBASE-15354 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Attachments: HBASE-15354-V0.patch, HBASE-15354-V1.patch, > HBASE-15354-V2.patch, HBASE-15354-V3.patch, HBASE-15354-V4.patch, > HBASE-15354-V5.patch, HBASE-15354-v5-addendum.patch > > > Currently we do not clear/update meta cache for some special exceptions if > the operation went through AsyncProcess#submit like HTable#put calls. But, we > clear meta cache without checking for these special exceptions in case of > other operations like gets, deletes etc because they directly go through the > RpcRetryingCaller#callWithRetries instead of the AsyncProcess. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15398) Cells loss or disorder when using family essential filter and partial scanning protocol
[
https://issues.apache.org/jira/browse/HBASE-15398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184224#comment-15184224
]
Phil Yang commented on HBASE-15398:
---
{quote}
I don't see new exception added which would alert the user that partial result
wouldn't be delivered.
{quote}
I haven't added this logic, should the exception be thrown in server or client?
> Cells loss or disorder when using family essential filter and partial
> scanning protocol
> ---
>
> Key: HBASE-15398
> URL: https://issues.apache.org/jira/browse/HBASE-15398
> Project: HBase
> Issue Type: Bug
> Components: dataloss, Scanners
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Phil Yang
>Assignee: Phil Yang
>Priority: Critical
> Attachments: 15398-test.txt, HBASE-15398.v1.txt
>
>
> In RegionScannerImpl, we have two heaps, storeHeap and joinedHeap. If we have
> a filter and it doesn't apply to all cf, the stores whose families needn't be
> filtered will be in joinedHeap. We scan storeHeap first, then joinedHeap,
> and merge the results and sort and return to client. We need sort because the
> order of Cell is rowkey/cf/cq/ts and a smaller cf may be in the joinedHeap.
> However, after HBASE-11544 we may transfer partial results when we get
> SIZE_LIMIT_REACHED_MID_ROW or other similar states. We may return a larger cf
> first because it is in storeHeap and then a smaller cf because it is in
> joinedHeap. Server won't hold all cells in a row and client doesn't have a
> sorting logic. The order of cf in Result for user is wrong.
> And a more critical bug is, if we get a LIMIT_REACHED_MID_ROW on the last
> cell of a row in storeHeap, we will break scanning in RegionScannerImpl and
> in populateResult we will change the state to SIZE_LIMIT_REACHED because next
> peeked cell is next row. But this is only the last cell of one and we have
> two... And SIZE_LIMIT_REACHED means this Result is not partial (by
> ScannerContext.partialResultFormed), client will see it and merge them and
> return to user with losing data of joinedHeap. On next scan we will read next
> row of storeHeap and joinedHeap is forgotten and never be read...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15400) Use DateTieredCompactor for Date Tiered Compaction
[
https://issues.apache.org/jira/browse/HBASE-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184215#comment-15184215
]
Duo Zhang commented on HBASE-15400:
---
[~clarax98007] I haven't include {{DateTieredCompactionRequest}} in my patch.
Instead, the {{DateTieredCompactor}} accepts a {{CompactionRequest}} and a
{{List}} (as boundaries). So if you do not want multiple output, you
could pass a boundaries only contains {{Long.MIN_VALUE}} and
{{Long.MAX_VALUE}}, the compactor will only output one file for you.
Thanks.
> Use DateTieredCompactor for Date Tiered Compaction
> --
>
> Key: HBASE-15400
> URL: https://issues.apache.org/jira/browse/HBASE-15400
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction
>Reporter: Clara Xiong
>Assignee: Clara Xiong
> Fix For: 2.0.0
>
> Attachments: HBASE-15400.patch
>
>
> When we compact, we can output multiple files along the current window
> boundaries. There are two use cases:
> 1. Major compaction: We want to output date tiered store files with data
> older than max age archived in trunks of the window size on the higher tier.
> 2. Bulk load files and the old file generated by major compaction before
> upgrading to DTCP.
> Pros:
> 1. Restore locality, process versioning, updates and deletes while
> maintaining the tiered layout.
> 2. The best way to fix a skewed layout.
>
> This work is based on a prototype of DateTieredCompactor from HBASE-15389 and
> focused on the part to meet need for these two use cases while supporting
> others. I have to call out a few design decisions:
> 1. We only want to output the files along all windows for major compaction.
> And we want to output multiple files older than max age in the trunks of the
> maximum tier window size determined by base window size, windows per tier and
> max age.
> 2. For minor compaction, we don't want to output too many files, which will
> remain around because of current restriction of contiguous compaction by seq
> id. I will only output two files if all the files in the windows are being
> combined, one for the data within window and the other for the out-of-window
> tail. If there is any file in the window excluded from compaction, only one
> file will be output from compaction. When the windows are promoted, the
> situation of out of order data will gradually improve. For the incoming
> window, we need to accommodate the case with user-specified future data.
> 3. We have to pass the boundaries with the list of store file as a complete
> time snapshot instead of two separate calls because window layout is
> determined by the time the computation is called. So we will need new type of
> compaction request.
> 4. Since we will assign the same seq id for all output files, we need to sort
> by maxTimestamp subsequently. Right now all compaction policy gets the files
> sorted for StoreFileManager which sorts by seq id and other criteria. I will
> use this order for DTCP only, to avoid impacting other compaction policies.
> 5. We need some cleanup of current design of StoreEngine and CompactionPolicy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15389) Write out multiple files when compaction
[
https://issues.apache.org/jira/browse/HBASE-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184201#comment-15184201
]
Duo Zhang commented on HBASE-15389:
---
AFAICT, {{CompactionPolicy}} is only used to select files for compacting. So if
you want to change the output of a compaction result, you need to implement a
{{Compactor}}. Here we only introduce a new {{Compactor}}, the new
{{StoreEngine}} will be introduced in HBASE-15400.
Thanks.
> Write out multiple files when compaction
>
>
> Key: HBASE-15389
> URL: https://issues.apache.org/jira/browse/HBASE-15389
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction
>Affects Versions: 2.0.0, 1.3.0, 0.98.19
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.3.0, 0.98.19
>
> Attachments: HBASE-15389-uc.patch, HBASE-15389-v1.patch,
> HBASE-15389-v2.patch, HBASE-15389.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184159#comment-15184159
]
Mikhail Antonov commented on HBASE-15406:
-
ephemeral znodes sound good to me.
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
