[jira] [Commented] (HBASE-15453) [Performance] Considering reverting HBASE-10015 - reinstance synchronized in StoreScanner
[ https://issues.apache.org/jira/browse/HBASE-15453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15194725#comment-15194725 ] Lars Hofhansl commented on HBASE-15453: --- For a Phoenix count\(*) query on the JDK8 machine I measure a 7.5% end-to-end improvement. I also tried the new JDK8 StampedLock, which should be better than ReentrantLock especially when not contended... Came to about the same as the intrinsic locks. > [Performance] Considering reverting HBASE-10015 - reinstance synchronized in > StoreScanner > - > > Key: HBASE-15453 > URL: https://issues.apache.org/jira/browse/HBASE-15453 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Attachments: 15453-0.98.txt > > > In HBASE-10015 back then I found that intrinsic locks (synchronized) in > StoreScanner are slower that explicit locks. > I was surprised by this. To make sure I added a simple perf test and many > folks ran it on their machines. All found that explicit locks were faster. > Now... I just ran that test again. On the latest JDK8 I find that now the > intrinsic locks are significantly faster: > (OpenJDK Runtime Environment (build 1.8.0_72-b15)) > Explicit locks: > 10 runs mean:2223.6 sigma:72.29412147609237 > Intrinsic locks: > 10 runs mean:1865.3 sigma:32.63755505548784 > I confirmed the same with timing some Phoenix scans. We can save a bunch of > time by changing this back > Arrghhh... So maybe it's time to revert this now...? > (Note that in trunk due to [~ram_krish]'s work, we do not lock in > StoreScanner anymore) > I'll attach the perf test and a patch that changes lock to synchronized, if > some folks could run this on 0.98, that'd be great. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15453) [Performance] Considering reverting HBASE-10015 - reinstate synchronized in StoreScanner
[ https://issues.apache.org/jira/browse/HBASE-15453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-15453: -- Summary: [Performance] Considering reverting HBASE-10015 - reinstate synchronized in StoreScanner (was: [Performance] Considering reverting HBASE-10015 - reinstance synchronized in StoreScanner) > [Performance] Considering reverting HBASE-10015 - reinstate synchronized in > StoreScanner > > > Key: HBASE-15453 > URL: https://issues.apache.org/jira/browse/HBASE-15453 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Attachments: 15453-0.98.txt > > > In HBASE-10015 back then I found that intrinsic locks (synchronized) in > StoreScanner are slower that explicit locks. > I was surprised by this. To make sure I added a simple perf test and many > folks ran it on their machines. All found that explicit locks were faster. > Now... I just ran that test again. On the latest JDK8 I find that now the > intrinsic locks are significantly faster: > (OpenJDK Runtime Environment (build 1.8.0_72-b15)) > Explicit locks: > 10 runs mean:2223.6 sigma:72.29412147609237 > Intrinsic locks: > 10 runs mean:1865.3 sigma:32.63755505548784 > I confirmed the same with timing some Phoenix scans. We can save a bunch of > time by changing this back > Arrghhh... So maybe it's time to revert this now...? > (Note that in trunk due to [~ram_krish]'s work, we do not lock in > StoreScanner anymore) > I'll attach the perf test and a patch that changes lock to synchronized, if > some folks could run this on 0.98, that'd be great. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8458) Support for batch version of checkAndPut() and checkAndDelete()
[ https://issues.apache.org/jira/browse/HBASE-8458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198531#comment-15198531 ] Lars Hofhansl commented on HBASE-8458: -- [~ekundin] which version of HBase are you using. I do agree with previous comments that no bulk api for CheckAndXXX is a bug. > Support for batch version of checkAndPut() and checkAndDelete() > --- > > Key: HBASE-8458 > URL: https://issues.apache.org/jira/browse/HBASE-8458 > Project: HBase > Issue Type: Improvement > Components: Client, regionserver >Affects Versions: 0.95.0 >Reporter: Hari Mankude > > The use case is that the user has multiple threads loading hundreds of keys > into a hbase table. Occasionally there are collisions in the keys being > uploaded by different threads. So for correctness, it is required to do > checkAndPut() instead of a put(). However, doing a checkAndPut() rpc for > every key update is non optimal. It would be good to have a batch version of > checkAndPut() similar to batch put(). The client can partition the keys on > region boundaries. > The jira is NOT looking for any type of cross-row locking or multi-row > atomicity with checkAndPut() > Batch version of checkAndDelete() is a similar requirement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15452) Consider removing checkScanOrder from StoreScanner.next
[ https://issues.apache.org/jira/browse/HBASE-15452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15197603#comment-15197603 ] Lars Hofhansl commented on HBASE-15452: --- Assertion were enabled in all cases. > Consider removing checkScanOrder from StoreScanner.next > --- > > Key: HBASE-15452 > URL: https://issues.apache.org/jira/browse/HBASE-15452 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Attachments: 15452-0.98.txt > > > In looking why we spent so much time in StoreScanner.next when doing a simple > Phoenix count\(*) query I came across checkScanOrder. Not only is this a > function dispatch (that the JIT would eventually inline), it also requires > setting the prevKV member for every Cell encountered. > Removing that logic a yields measurable end-to-end improvement of 5-20% (in > 0.98). > I will repeat this test on my work machine tomorrow. > I think we're stable enough to remove that check anyway. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15479) No more garbage or beware of autoboxing
[ https://issues.apache.org/jira/browse/HBASE-15479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15202988#comment-15202988 ] Lars Hofhansl commented on HBASE-15479: --- Is -v2 complete? I see the signature change for canTakeOperation, but no caller change in the patch. > No more garbage or beware of autoboxing > --- > > Key: HBASE-15479 > URL: https://issues.apache.org/jira/browse/HBASE-15479 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-15479-v1.patch, HBASE-15479-v2.patch > > > Quick journey with JMC in a profile mode revealed very interesting and > unexpected heap polluter on a client side. Patch will shortly follow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15479) No more garbage or beware of autoboxing
[ https://issues.apache.org/jira/browse/HBASE-15479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203016#comment-15203016 ] Lars Hofhansl commented on HBASE-15479: --- Ahh... The call is not in the patch but the parameter change is. Cool. Seems good to commit to all branches. > No more garbage or beware of autoboxing > --- > > Key: HBASE-15479 > URL: https://issues.apache.org/jira/browse/HBASE-15479 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-15479-v1.patch, HBASE-15479-v2.patch > > > Quick journey with JMC in a profile mode revealed very interesting and > unexpected heap polluter on a client side. Patch will shortly follow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15392) Single Cell Get reads two HFileBlocks
[
https://issues.apache.org/jira/browse/HBASE-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203085#comment-15203085
]
Lars Hofhansl commented on HBASE-15392:
---
[~stack], let's also change the compare condition to > rather than >= (as
[~ram_krish] suggests). It won't make any difference but it will be easier to
understand in the future I feel.
> Single Cell Get reads two HFileBlocks
> -
>
> Key: HBASE-15392
> URL: https://issues.apache.org/jira/browse/HBASE-15392
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: stack
>Assignee: stack
> Attachments: 15392-0.98-looksee.txt, 15392.wip.patch,
> 15392v2.wip.patch, 15392v3.wip.patch, 15392v4.patch, 15392v5.patch,
> 15392v6.patch, 15392v7 (1).patch, 15392v7.patch, 15392v7.patch,
> 15392v7.patch, 15392v7.patch, 15392v7.patch, HBASE-15392_suggest.patch,
> gc.png, gc.png, io.png, no_optimize.patch, no_optimize.patch, reads.png,
> reads.png, two_seeks.txt
>
>
> As found by Daniel "SystemTap" Pol, a simple Get results in our reading two
> HFileBlocks, the one that contains the wanted Cell, and the block that
> follows.
> Here is a bit of custom logging that logs a stack trace on each HFileBlock
> read so you can see the call stack responsible:
> {code}
> 2016-03-03 22:20:30,191 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] regionserver.StoreScanner:
> START LOOP
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] regionserver.StoreScanner:
> QCODE SEEK_NEXT_COL
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileBlockIndex:
> STARTED WHILE
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.CombinedBlockCache:
> OUT OF L2
> 2016-03-03 22:20:30,192 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] bucket.BucketCache: Read
> offset=31409152, len=2103
> 2016-03-03 22:20:30,192 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] bucket.FileIOEngine:
> offset=31409152, length=2103
> 2016-03-03 22:20:30,193 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileReaderImpl:
> From Cache [blockType=DATA, fileOffset=2055421, headerSize=33,
> onDiskSizeWithoutHeader=2024, uncompressedSizeWithoutHeader=2020,
> prevBlockOffset=2053364, isUseHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, onDiskDataSizeWithHeader=2053,
> getOnDiskSizeWithHeader=2057, totalChecksumBytes=4, isUnpacked=true,
> buf=[org.apache.hadoop.hbase.nio.SingleByteBuff@e19fbd54],
> dataBeginsWith=\x00\x00\x00)\x00\x00\x01`\x00\x16user995139035672819231,
> fileContext=[usesHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true,
> includesTags=false, compressAlgo=NONE, compressTags=false,
> cryptoContext=[cipher=NONE keyHash=NONE]]]
> 2016-03-03 22:20:30,193 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileReaderImpl:
> Cache hit return [blockType=DATA, fileOffset=2055421, headerSize=33,
> onDiskSizeWithoutHeader=2024, uncompressedSizeWithoutHeader=2020,
> prevBlockOffset=2053364, isUseHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, onDiskDataSizeWithHeader=2053,
> getOnDiskSizeWithHeader=2057, totalChecksumBytes=4, isUnpacked=true,
> buf=[org.apache.hadoop.hbase.nio.SingleByteBuff@e19fbd54],
> dataBeginsWith=\x00\x00\x00)\x00\x00\x01`\x00\x16user995139035672819231,
> fileContext=[usesHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true,
> includesTags=false, compressAlgo=NONE, compressTags=false,
> cryptoContext=[cipher=NONE keyHash=NONE]]]
> java.lang.Throwable
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1515)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$CellBasedKeyBlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:324)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:831)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.reseekTo(HFileReaderImpl.java:812)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:288)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:198)
> at
> org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:321)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:279)
> at
[jira] [Commented] (HBASE-12148) Remove TimeRangeTracker as point of contention when many threads writing a Store
[ https://issues.apache.org/jira/browse/HBASE-12148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15201088#comment-15201088 ] Lars Hofhansl commented on HBASE-12148: --- I see. So this is at _read_ time from a StoreFile. Is that the main problem? Not contention at write time. Is the problem that readers lock each other out? That we can solve with a simple ReadWrite lock. (but that still takes a memory fence) Or we can have two TimeRangeTracker implementations. One that mutable (and synchronized) for the memstore, and one that is immutable (and not synchronized at all) for StoreFiles. > Remove TimeRangeTracker as point of contention when many threads writing a > Store > > > Key: HBASE-12148 > URL: https://issues.apache.org/jira/browse/HBASE-12148 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0, 0.99.1 >Reporter: stack >Assignee: Walter Koetke > Fix For: 2.0.0, 1.3.0, 0.98.19 > > Attachments: > 0001-In-AtomicUtils-change-updateMin-and-updateMax-to-ret.patch, > 12148.addendum.txt, 12148.txt, 12148.txt, 12148v2.txt, 12148v2.txt, > HBASE-12148-V3.patch, HBASE-12148.txt, HBASE-12148V2.txt, Screen Shot > 2014-10-01 at 3.39.46 PM.png, Screen Shot 2014-10-01 at 3.41.07 PM.png, > TimeRangeTracker.tiff > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12148) Remove TimeRangeTracker as point of contention when many threads writing a Store
[ https://issues.apache.org/jira/browse/HBASE-12148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200977#comment-15200977 ] Lars Hofhansl commented on HBASE-12148: --- Do we see actual contention, as in threads waiting for each other? Or is this slow down due to memory barrier implied by a synchronized (and lock, and CAS, and volatile, and AtomicXXX, etc)? [~stack], [~wkoetke], where does this show as a problem? It is at the time when write a store file? Or is it at the time when we insert into the memstore? > Remove TimeRangeTracker as point of contention when many threads writing a > Store > > > Key: HBASE-12148 > URL: https://issues.apache.org/jira/browse/HBASE-12148 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0, 0.99.1 >Reporter: stack >Assignee: Walter Koetke > Fix For: 2.0.0, 1.3.0, 0.98.19 > > Attachments: > 0001-In-AtomicUtils-change-updateMin-and-updateMax-to-ret.patch, > 12148.addendum.txt, 12148.txt, 12148.txt, 12148v2.txt, 12148v2.txt, > HBASE-12148-V3.patch, HBASE-12148.txt, HBASE-12148V2.txt, Screen Shot > 2014-10-01 at 3.39.46 PM.png, Screen Shot 2014-10-01 at 3.41.07 PM.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12148) Remove TimeRangeTracker as point of contention when many threads writing a Store
[ https://issues.apache.org/jira/browse/HBASE-12148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204915#comment-15204915 ] Lars Hofhansl commented on HBASE-12148: --- If this read load you're applying? Or mixed read-write load? If the problem is actual contention (threads lock each other out and get suspended) then we should be able to help with read/write locks or mutable/immutable tracker as I suggest above. > Remove TimeRangeTracker as point of contention when many threads writing a > Store > > > Key: HBASE-12148 > URL: https://issues.apache.org/jira/browse/HBASE-12148 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0, 0.99.1 >Reporter: stack >Assignee: Walter Koetke > Fix For: 2.0.0, 1.3.0, 0.98.19 > > Attachments: > 0001-In-AtomicUtils-change-updateMin-and-updateMax-to-ret.patch, > 12148.addendum.txt, 12148.txt, 12148.txt, 12148v2.txt, 12148v2.txt, > HBASE-12148-V3.patch, HBASE-12148.txt, HBASE-12148V2.txt, Screen Shot > 2014-10-01 at 3.39.46 PM.png, Screen Shot 2014-10-01 at 3.41.07 PM.png, > TimeRangeTracker.tiff > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14613) Remove MemStoreChunkPool?
[ https://issues.apache.org/jira/browse/HBASE-14613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15207757#comment-15207757 ] Lars Hofhansl commented on HBASE-14613: --- Not sure what you are referring to. SLAB making more garbage, or absence of SLAB generating more garbage? > Remove MemStoreChunkPool? > - > > Key: HBASE-14613 > URL: https://issues.apache.org/jira/browse/HBASE-14613 > Project: HBase > Issue Type: Brainstorming >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 14613-0.98.txt, gc.png, writes.png > > > I just stumbled across MemStoreChunkPool. The idea behind is to reuse chunks > of allocations rather than letting the GC handle this. > Now, it's off by default, and it seems to me to be of dubious value. I'd > recommend just removing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15392) Single Cell Get reads two HFileBlocks
[
https://issues.apache.org/jira/browse/HBASE-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15207841#comment-15207841
]
Lars Hofhansl commented on HBASE-15392:
---
-v8 is something different. Gigantic 311KB patch. Wrong Jira?!
> Single Cell Get reads two HFileBlocks
> -
>
> Key: HBASE-15392
> URL: https://issues.apache.org/jira/browse/HBASE-15392
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: stack
>Assignee: stack
> Attachments: 15392-0.98-looksee.txt, 15392.wip.patch,
> 15392v2.wip.patch, 15392v3.wip.patch, 15392v4.patch, 15392v5.patch,
> 15392v6.patch, 15392v7 (1).patch, 15392v7.patch, 15392v7.patch,
> 15392v7.patch, 15392v7.patch, 15392v7.patch, HBASE-15265-v8.patch,
> HBASE-15392_suggest.patch, gc.png, gc.png, io.png, no_optimize.patch,
> no_optimize.patch, reads.png, reads.png, two_seeks.txt
>
>
> As found by Daniel "SystemTap" Pol, a simple Get results in our reading two
> HFileBlocks, the one that contains the wanted Cell, and the block that
> follows.
> Here is a bit of custom logging that logs a stack trace on each HFileBlock
> read so you can see the call stack responsible:
> {code}
> 2016-03-03 22:20:30,191 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] regionserver.StoreScanner:
> START LOOP
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] regionserver.StoreScanner:
> QCODE SEEK_NEXT_COL
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileBlockIndex:
> STARTED WHILE
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.CombinedBlockCache:
> OUT OF L2
> 2016-03-03 22:20:30,192 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] bucket.BucketCache: Read
> offset=31409152, len=2103
> 2016-03-03 22:20:30,192 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] bucket.FileIOEngine:
> offset=31409152, length=2103
> 2016-03-03 22:20:30,193 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileReaderImpl:
> From Cache [blockType=DATA, fileOffset=2055421, headerSize=33,
> onDiskSizeWithoutHeader=2024, uncompressedSizeWithoutHeader=2020,
> prevBlockOffset=2053364, isUseHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, onDiskDataSizeWithHeader=2053,
> getOnDiskSizeWithHeader=2057, totalChecksumBytes=4, isUnpacked=true,
> buf=[org.apache.hadoop.hbase.nio.SingleByteBuff@e19fbd54],
> dataBeginsWith=\x00\x00\x00)\x00\x00\x01`\x00\x16user995139035672819231,
> fileContext=[usesHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true,
> includesTags=false, compressAlgo=NONE, compressTags=false,
> cryptoContext=[cipher=NONE keyHash=NONE]]]
> 2016-03-03 22:20:30,193 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileReaderImpl:
> Cache hit return [blockType=DATA, fileOffset=2055421, headerSize=33,
> onDiskSizeWithoutHeader=2024, uncompressedSizeWithoutHeader=2020,
> prevBlockOffset=2053364, isUseHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, onDiskDataSizeWithHeader=2053,
> getOnDiskSizeWithHeader=2057, totalChecksumBytes=4, isUnpacked=true,
> buf=[org.apache.hadoop.hbase.nio.SingleByteBuff@e19fbd54],
> dataBeginsWith=\x00\x00\x00)\x00\x00\x01`\x00\x16user995139035672819231,
> fileContext=[usesHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true,
> includesTags=false, compressAlgo=NONE, compressTags=false,
> cryptoContext=[cipher=NONE keyHash=NONE]]]
> java.lang.Throwable
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1515)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$CellBasedKeyBlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:324)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:831)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.reseekTo(HFileReaderImpl.java:812)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:288)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:198)
> at
> org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:321)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:279)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:806)
> at
> org.a
[jira] [Commented] (HBASE-15392) Single Cell Get reads two HFileBlocks
[
https://issues.apache.org/jira/browse/HBASE-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15207842#comment-15207842
]
Lars Hofhansl commented on HBASE-15392:
---
Yep.
> Single Cell Get reads two HFileBlocks
> -
>
> Key: HBASE-15392
> URL: https://issues.apache.org/jira/browse/HBASE-15392
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: stack
>Assignee: stack
> Attachments: 15392-0.98-looksee.txt, 15392.wip.patch,
> 15392v2.wip.patch, 15392v3.wip.patch, 15392v4.patch, 15392v5.patch,
> 15392v6.patch, 15392v7 (1).patch, 15392v7.patch, 15392v7.patch,
> 15392v7.patch, 15392v7.patch, 15392v7.patch, HBASE-15265-v8.patch,
> HBASE-15392_suggest.patch, gc.png, gc.png, io.png, no_optimize.patch,
> no_optimize.patch, reads.png, reads.png, two_seeks.txt
>
>
> As found by Daniel "SystemTap" Pol, a simple Get results in our reading two
> HFileBlocks, the one that contains the wanted Cell, and the block that
> follows.
> Here is a bit of custom logging that logs a stack trace on each HFileBlock
> read so you can see the call stack responsible:
> {code}
> 2016-03-03 22:20:30,191 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] regionserver.StoreScanner:
> START LOOP
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] regionserver.StoreScanner:
> QCODE SEEK_NEXT_COL
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileBlockIndex:
> STARTED WHILE
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.CombinedBlockCache:
> OUT OF L2
> 2016-03-03 22:20:30,192 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] bucket.BucketCache: Read
> offset=31409152, len=2103
> 2016-03-03 22:20:30,192 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] bucket.FileIOEngine:
> offset=31409152, length=2103
> 2016-03-03 22:20:30,193 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileReaderImpl:
> From Cache [blockType=DATA, fileOffset=2055421, headerSize=33,
> onDiskSizeWithoutHeader=2024, uncompressedSizeWithoutHeader=2020,
> prevBlockOffset=2053364, isUseHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, onDiskDataSizeWithHeader=2053,
> getOnDiskSizeWithHeader=2057, totalChecksumBytes=4, isUnpacked=true,
> buf=[org.apache.hadoop.hbase.nio.SingleByteBuff@e19fbd54],
> dataBeginsWith=\x00\x00\x00)\x00\x00\x01`\x00\x16user995139035672819231,
> fileContext=[usesHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true,
> includesTags=false, compressAlgo=NONE, compressTags=false,
> cryptoContext=[cipher=NONE keyHash=NONE]]]
> 2016-03-03 22:20:30,193 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileReaderImpl:
> Cache hit return [blockType=DATA, fileOffset=2055421, headerSize=33,
> onDiskSizeWithoutHeader=2024, uncompressedSizeWithoutHeader=2020,
> prevBlockOffset=2053364, isUseHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, onDiskDataSizeWithHeader=2053,
> getOnDiskSizeWithHeader=2057, totalChecksumBytes=4, isUnpacked=true,
> buf=[org.apache.hadoop.hbase.nio.SingleByteBuff@e19fbd54],
> dataBeginsWith=\x00\x00\x00)\x00\x00\x01`\x00\x16user995139035672819231,
> fileContext=[usesHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true,
> includesTags=false, compressAlgo=NONE, compressTags=false,
> cryptoContext=[cipher=NONE keyHash=NONE]]]
> java.lang.Throwable
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1515)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$CellBasedKeyBlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:324)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:831)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.reseekTo(HFileReaderImpl.java:812)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:288)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:198)
> at
> org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:321)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:279)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:806)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekAsDirecti
[jira] [Commented] (HBASE-14613) Remove MemStoreChunkPool?
[ https://issues.apache.org/jira/browse/HBASE-14613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15207901#comment-15207901 ] Lars Hofhansl commented on HBASE-14613: --- [~vik.karma], FYI. Interesting bit here about disabling the memstore SLAB allocator when using G1. > Remove MemStoreChunkPool? > - > > Key: HBASE-14613 > URL: https://issues.apache.org/jira/browse/HBASE-14613 > Project: HBase > Issue Type: Brainstorming >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 14613-0.98.txt, gc.png, writes.png > > > I just stumbled across MemStoreChunkPool. The idea behind is to reuse chunks > of allocations rather than letting the GC handle this. > Now, it's off by default, and it seems to me to be of dubious value. I'd > recommend just removing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12148) Remove TimeRangeTracker as point of contention when many threads writing a Store
[ https://issues.apache.org/jira/browse/HBASE-12148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15208954#comment-15208954 ] Lars Hofhansl commented on HBASE-12148: --- So what do we do here?! > Remove TimeRangeTracker as point of contention when many threads writing a > Store > > > Key: HBASE-12148 > URL: https://issues.apache.org/jira/browse/HBASE-12148 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0, 0.99.1 >Reporter: stack >Assignee: Walter Koetke > Fix For: 2.0.0, 1.3.0, 0.98.19 > > Attachments: > 0001-In-AtomicUtils-change-updateMin-and-updateMax-to-ret.patch, > 12148.addendum.txt, 12148.txt, 12148.txt, 12148v2.txt, 12148v2.txt, > HBASE-12148-V3.patch, HBASE-12148.txt, HBASE-12148V2.txt, Screen Shot > 2014-10-01 at 3.39.46 PM.png, Screen Shot 2014-10-01 at 3.41.07 PM.png, > TimeRangeTracker.tiff > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15453) [Performance] Considering reverting HBASE-10015 - reinstate synchronized in StoreScanner
[ https://issues.apache.org/jira/browse/HBASE-15453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15208984#comment-15208984 ] Lars Hofhansl commented on HBASE-15453: --- So what do we do here? I'll do one more test later... See how it compares to no memory fencing at all. So perf loss on JDK 7 is significant, so is the perf gain with JDK 8. > [Performance] Considering reverting HBASE-10015 - reinstate synchronized in > StoreScanner > > > Key: HBASE-15453 > URL: https://issues.apache.org/jira/browse/HBASE-15453 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Attachments: 15453-0.98.txt > > > In HBASE-10015 back then I found that intrinsic locks (synchronized) in > StoreScanner are slower that explicit locks. > I was surprised by this. To make sure I added a simple perf test and many > folks ran it on their machines. All found that explicit locks were faster. > Now... I just ran that test again. On the latest JDK8 I find that now the > intrinsic locks are significantly faster: > (OpenJDK Runtime Environment (build 1.8.0_72-b15)) > Explicit locks: > 10 runs mean:2223.6 sigma:72.29412147609237 > Intrinsic locks: > 10 runs mean:1865.3 sigma:32.63755505548784 > I confirmed the same with timing some Phoenix scans. We can save a bunch of > time by changing this back > Arrghhh... So maybe it's time to revert this now...? > (Note that in trunk due to [~ram_krish]'s work, we do not lock in > StoreScanner anymore) > I'll attach the perf test and a patch that changes lock to synchronized, if > some folks could run this on 0.98, that'd be great. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14613) Remove MemStoreChunkPool?
[ https://issues.apache.org/jira/browse/HBASE-14613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15212846#comment-15212846 ] Lars Hofhansl commented on HBASE-14613: --- [~apurtell], [~vik.karma], maybe we should redo our G1 testing with this in mind. > Remove MemStoreChunkPool? > - > > Key: HBASE-14613 > URL: https://issues.apache.org/jira/browse/HBASE-14613 > Project: HBase > Issue Type: Brainstorming >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 14613-0.98.txt, gc.png, writes.png > > > I just stumbled across MemStoreChunkPool. The idea behind is to reuse chunks > of allocations rather than letting the GC handle this. > Now, it's off by default, and it seems to me to be of dubious value. I'd > recommend just removing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14613) Remove MemStoreChunkPool?
[ https://issues.apache.org/jira/browse/HBASE-14613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15212845#comment-15212845 ] Lars Hofhansl commented on HBASE-14613: --- Didn't think my musing about G1 and MSLAB would lead to a long a useful discussion :) Should we set InitiatingHeapOccupancyPercent based on configuration? By default we allow 40% of the heap for block cache and 40% for the memstore. When configured such, should we set InitiatingHeapOccupancyPercent to 80? Or maybe if we assume the memstores are on average 1/2 filled maybe this reasoning would lead to 60% as a good default...? > Remove MemStoreChunkPool? > - > > Key: HBASE-14613 > URL: https://issues.apache.org/jira/browse/HBASE-14613 > Project: HBase > Issue Type: Brainstorming >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 14613-0.98.txt, gc.png, writes.png > > > I just stumbled across MemStoreChunkPool. The idea behind is to reuse chunks > of allocations rather than letting the GC handle this. > Now, it's off by default, and it seems to me to be of dubious value. I'd > recommend just removing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15506) FSDataOutputStream.write() allocates new byte buffer on each operation
[ https://issues.apache.org/jira/browse/HBASE-15506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15213853#comment-15213853 ] Lars Hofhansl commented on HBASE-15506: --- Let's be careful, please. Many times when one tries to be smarter than the GC it backfires (false lifetimes of reused objects, etc). Allocating stuff on the heap is not necessary bad by itself. That's what the heap for. :) It's only bad when there's a _measurable_ performance impact (including long GC pauses)... Or if the allocation is strictly unnecessary. Which of the two option is using more memory bandwidth? That's the metric I'd be interested in. In any case, does this hold for G1 as well? It's possible I am missing something. We're talking about 64KB buffers allocating for the HDFS stream packets, right? 100MB of garbage in 64KB chunks is a mere 1600 objects. Surely we're not going to be worried about that. Are compactions or HFileWriter doing something stupid? Since there is fix in HDFS-7276, I clearly must be missing something. But what is that? > FSDataOutputStream.write() allocates new byte buffer on each operation > -- > > Key: HBASE-15506 > URL: https://issues.apache.org/jira/browse/HBASE-15506 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > Deep inside stack trace in DFSOutputStream.createPacket. > This should be opened in HDFS. This JIRA is to track HDFS work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15506) FSDataOutputStream.write() allocates new byte buffer on each operation
[ https://issues.apache.org/jira/browse/HBASE-15506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15217374#comment-15217374 ] Lars Hofhansl commented on HBASE-15506: --- My point is that that's a problem as such. That's how Java is designed to work. It's only a problem when it is problem :) As in, it definitely slows things down, causes long GC pauses, etc. I don't have to tell you, but for just completeness of the discussion here: The principle costs to the garbage collector are (1) tracing all objects from the "root" objects and (2) collecting all unreachable objects. Obviously #1 is expensive when many objects need to be traced, and #2 is expensive when objects have to be moved (for example to reduce memory fragmentation) 64KB objects do not worry me, even if we have many GBs of them, it's just not many references to track. Further since they are all the same size, we won't fragment the heap in bad ways. Reusing objects (IMHO) is simply a very questionable technique. Especially when you have to reset the objects, which is more expensive then fitting a new object into a free slot of the same size. For what's it's worth, I have seen bad behaviour during heavy loading phases. I was always able to configure the GC accordingly, though. In any case, we should be able to create some single server test workload that exhibits problems if there are any. Those are good tests to have anyway, not just a way to appease me. > FSDataOutputStream.write() allocates new byte buffer on each operation > -- > > Key: HBASE-15506 > URL: https://issues.apache.org/jira/browse/HBASE-15506 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > Deep inside stack trace in DFSOutputStream.createPacket. > This should be opened in HDFS. This JIRA is to track HDFS work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-15506) FSDataOutputStream.write() allocates new byte buffer on each operation
[ https://issues.apache.org/jira/browse/HBASE-15506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15217374#comment-15217374 ] Lars Hofhansl edited comment on HBASE-15506 at 3/30/16 4:51 AM: My point is that that's not a problem as such. That's how Java is designed to work. It's only a problem when it is problem :) As in, it definitely slows things down, causes long GC pauses, etc. I don't have to tell you, but for just completeness of the discussion here: The principle costs to the garbage collector are (1) tracing all objects from the "root" objects and (2) collecting all unreachable objects. Obviously #1 is expensive when many objects need to be traced, and #2 is expensive when objects have to be moved (for example to reduce memory fragmentation) 64KB objects do not worry me, even if we have many GBs of them, it's just not many references to track. Further since they are all the same size, we won't fragment the heap in bad ways. Reusing objects (IMHO) is simply a very questionable technique. Especially when you have to reset the objects, which is more expensive then fitting a new object into a free slot of the same size. For what's it's worth, I have seen bad behaviour during heavy loading phases. I was always able to configure the GC accordingly, though. In any case, we should be able to create some single server test workload that exhibits problems if there are any. Those are good tests to have anyway, not just a way to appease me. was (Author: lhofhansl): My point is that that's a problem as such. That's how Java is designed to work. It's only a problem when it is problem :) As in, it definitely slows things down, causes long GC pauses, etc. I don't have to tell you, but for just completeness of the discussion here: The principle costs to the garbage collector are (1) tracing all objects from the "root" objects and (2) collecting all unreachable objects. Obviously #1 is expensive when many objects need to be traced, and #2 is expensive when objects have to be moved (for example to reduce memory fragmentation) 64KB objects do not worry me, even if we have many GBs of them, it's just not many references to track. Further since they are all the same size, we won't fragment the heap in bad ways. Reusing objects (IMHO) is simply a very questionable technique. Especially when you have to reset the objects, which is more expensive then fitting a new object into a free slot of the same size. For what's it's worth, I have seen bad behaviour during heavy loading phases. I was always able to configure the GC accordingly, though. In any case, we should be able to create some single server test workload that exhibits problems if there are any. Those are good tests to have anyway, not just a way to appease me. > FSDataOutputStream.write() allocates new byte buffer on each operation > -- > > Key: HBASE-15506 > URL: https://issues.apache.org/jira/browse/HBASE-15506 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > Deep inside stack trace in DFSOutputStream.createPacket. > This should be opened in HDFS. This JIRA is to track HDFS work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15506) FSDataOutputStream.write() allocates new byte buffer on each operation
[ https://issues.apache.org/jira/browse/HBASE-15506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15219340#comment-15219340 ] Lars Hofhansl commented on HBASE-15506: --- nginx is implemented in C/C++. We can easily find about as many articles and examples for it as we can find them against it. I am only saying that a statement like "heap allocations are bad" is simply not generally true. Let's test, have some numbers, and I will happily shut up about it (in fact if I was wrong I will have learned something) > FSDataOutputStream.write() allocates new byte buffer on each operation > -- > > Key: HBASE-15506 > URL: https://issues.apache.org/jira/browse/HBASE-15506 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > Deep inside stack trace in DFSOutputStream.createPacket. > This should be opened in HDFS. This JIRA is to track HDFS work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-7912) HBase Backup/Restore Based on HBase Snapshot
[ https://issues.apache.org/jira/browse/HBASE-7912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15223137#comment-15223137 ] Lars Hofhansl commented on HBASE-7912: -- Pardon my ignorance here, but can somebody briefly explain the pros and cons of the two layout options? "It is as it is" but also "that's how the rest of the HBase does it" both do not really sound like compelling arguments to me :) Offhand it feels useful to group everything that belongs to a specific backup in a single directory tree and from that angle would favour leading with the backup ID... There might be various other reasons. > HBase Backup/Restore Based on HBase Snapshot > > > Key: HBASE-7912 > URL: https://issues.apache.org/jira/browse/HBASE-7912 > Project: HBase > Issue Type: Sub-task >Reporter: Richard Ding >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: HBaseBackupRestore-Jira-7912-DesignDoc-v1.pdf, > HBaseBackupRestore-Jira-7912-DesignDoc-v2.pdf, > HBaseBackupRestore-Jira-7912-v4.pdf, HBaseBackupRestore-Jira-7912-v5 .pdf, > HBaseBackupRestore-Jira-7912-v6.pdf, HBaseBackupandRestore.pdf, > HBase_BackupRestore-Jira-7912-CLI-v1.pdf > > > Finally, we completed the implementation of our backup/restore solution, and > would like to share with community through this jira. > We are leveraging existing hbase snapshot feature, and provide a general > solution to common users. Our full backup is using snapshot to capture > metadata locally and using exportsnapshot to move data to another cluster; > the incremental backup is using offline-WALplayer to backup HLogs; we also > leverage global distribution rolllog and flush to improve performance; other > added-on values such as convert, merge, progress report, and CLI commands. So > that a common user can backup hbase data without in-depth knowledge of hbase. > Our solution also contains some usability features for enterprise users. > The detail design document and CLI command will be attached in this jira. We > plan to use 10~12 subtasks to share each of the following features, and > document the detail implement in the subtasks: > * *Full Backup* : provide local and remote back/restore for a list of tables > * *offline-WALPlayer* to convert HLog to HFiles offline (for incremental > backup) > * *distributed* Logroll and distributed flush > * Backup *Manifest* and history > * *Incremental* backup: to build on top of full backup as daily/weekly backup > * *Convert* incremental backup WAL files into hfiles > * *Merge* several backup images into one(like merge weekly into monthly) > * *add and remove* table to and from Backup image > * *Cancel* a backup process > * backup progress *status* > * full backup based on *existing snapshot* > *-* > *Below is the original description, to keep here as the history for the > design and discussion back in 2013* > There have been attempts in the past to come up with a viable HBase > backup/restore solution (e.g., HBASE-4618). Recently, there are many > advancements and new features in HBase, for example, FileLink, Snapshot, and > Distributed Barrier Procedure. This is a proposal for a backup/restore > solution that utilizes these new features to achieve better performance and > consistency. > > A common practice of backup and restore in database is to first take full > baseline backup, and then periodically take incremental backup that capture > the changes since the full baseline backup. HBase cluster can store massive > amount data. Combination of full backups with incremental backups has > tremendous benefit for HBase as well. The following is a typical scenario > for full and incremental backup. > # The user takes a full backup of a table or a set of tables in HBase. > # The user schedules periodical incremental backups to capture the changes > from the full backup, or from last incremental backup. > # The user needs to restore table data to a past point of time. > # The full backup is restored to the table(s) or to different table name(s). > Then the incremental backups that are up to the desired point in time are > applied on top of the full backup. > We would support the following key features and capabilities. > * Full backup uses HBase snapshot to capture HFiles. > * Use HBase WALs to capture incremental changes, but we use bulk load of > HFiles for fast incremental restore. > * Support single table or a set of tables, and column family level backup and > restore. > * Restore to different table names. > * Support adding additional tables or CF to backup set without interruption > of incremental backup schedule. > * Supp
[jira] [Updated] (HBASE-15054) Allow 0.94 to compile with JDK8
[ https://issues.apache.org/jira/browse/HBASE-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-15054: -- Fix Version/s: 0.94.28 > Allow 0.94 to compile with JDK8 > --- > > Key: HBASE-15054 > URL: https://issues.apache.org/jira/browse/HBASE-15054 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.94.28 > > Attachments: 15054.txt > > > Fix the following two problems: > # PoolMap > # InputSampler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15059) Allow 0.94 to compile against Hadoop 2.7.x
[ https://issues.apache.org/jira/browse/HBASE-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15075517#comment-15075517 ] Lars Hofhansl commented on HBASE-15059: --- Thanks. What do you think about this change? Need to state somewhere that one needs to regenerate the protobuf code (and since I couldn't figure out how to do that, I added the older shell-script that master used to have). > Allow 0.94 to compile against Hadoop 2.7.x > -- > > Key: HBASE-15059 > URL: https://issues.apache.org/jira/browse/HBASE-15059 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Fix For: 0.94.28 > > Attachments: 15059-v2.txt, 15059.txt > > > Currently HBase 0.94 cannot be compiled against Hadoop 2.7. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15054) Allow 0.94 to compile with JDK8
[
https://issues.apache.org/jira/browse/HBASE-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-15054:
--
Release Note:
This change replaces
{{public boolean remove(K key, V value)}} in {{PoolMap}}
with
{{public boolean removeValue(K key, V value)}}
Direct users of PoolMap will need to change the code accordingly.
> Allow 0.94 to compile with JDK8
> ---
>
> Key: HBASE-15054
> URL: https://issues.apache.org/jira/browse/HBASE-15054
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.28
>
> Attachments: 15054.txt
>
>
> Fix the following two problems:
> # PoolMap
> # InputSampler
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15054) Allow 0.94 to compile with JDK8
[
https://issues.apache.org/jira/browse/HBASE-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-15054:
--
Release Note:
This change replaces
public boolean remove(K key, V value) in PoolMap
with
public boolean removeValue(K key, V value)
Direct users of PoolMap will need to change the code accordingly.
was:
This change replaces
{{public boolean remove(K key, V value)}} in {{PoolMap}}
with
{{public boolean removeValue(K key, V value)}}
Direct users of PoolMap will need to change the code accordingly.
> Allow 0.94 to compile with JDK8
> ---
>
> Key: HBASE-15054
> URL: https://issues.apache.org/jira/browse/HBASE-15054
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.28
>
> Attachments: 15054.txt
>
>
> Fix the following two problems:
> # PoolMap
> # InputSampler
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15059) Allow 0.94 to compile against Hadoop 2.7.x
[ https://issues.apache.org/jira/browse/HBASE-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15075526#comment-15075526 ] Lars Hofhansl commented on HBASE-15059: --- bq. Too big a change for point release. Will make it so can't run on hadoop 1s. Agreed. The patch does this now: * PB 2.4.0a is used by default. * If -Dhadoop.profile=2.7 is specified then PB 2.5 and commons.io 2.4 will be pulled in as dependencies * When building for later Hadoops the user has to manually execute dev-support.build-proto.sh, before building the rest of HBase Would be nice to automate the last part via maven, but I think for 0.94 that might be overkill. > Allow 0.94 to compile against Hadoop 2.7.x > -- > > Key: HBASE-15059 > URL: https://issues.apache.org/jira/browse/HBASE-15059 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Fix For: 0.94.28 > > Attachments: 15059-v2.txt, 15059.txt > > > Currently HBase 0.94 cannot be compiled against Hadoop 2.7. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14468) Compaction improvements: FIFO compaction policy
[
https://issues.apache.org/jira/browse/HBASE-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-14468:
--
Attachment: 14468-0.98.txt
For what's it worth, here's a 0.98 patch for your consideration. [~apurtell],
[~stack]
Changes to the 1.2 patch are cosmetic...
...Except for the test! It seems in 1.0 and later, the compact-after-flush
feature is broken. In 0.98 it is working, and hence the test fails, because all
hfiles are compacted away straight after each flush, so I had to change the
time handling to avoid that.
Something to look at at 1.2+
> Compaction improvements: FIFO compaction policy
> ---
>
> Key: HBASE-14468
> URL: https://issues.apache.org/jira/browse/HBASE-14468
> Project: HBase
> Issue Type: Improvement
> Components: Compaction, Performance
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14468-0.98.txt, HBASE-14468-v1.patch,
> HBASE-14468-v10.patch, HBASE-14468-v2.patch, HBASE-14468-v3.patch,
> HBASE-14468-v4.patch, HBASE-14468-v5.patch, HBASE-14468-v6.patch,
> HBASE-14468-v7.patch, HBASE-14468-v8.patch, HBASE-14468-v9.patch
>
>
> h2. FIFO Compaction
> h3. Introduction
> FIFO compaction policy selects only files which have all cells expired. The
> column family MUST have non-default TTL.
> Essentially, FIFO compactor does only one job: collects expired store files.
> These are some applications which could benefit the most:
> # Use it for very high volume raw data which has low TTL and which is the
> source of another data (after additional processing). Example: Raw
> time-series vs. time-based rollup aggregates and compacted time-series. We
> collect raw time-series and store them into CF with FIFO compaction policy,
> periodically we run task which creates rollup aggregates and compacts
> time-series, the original raw data can be discarded after that.
> # Use it for data which can be kept entirely in a a block cache (RAM/SSD).
> Say we have local SSD (1TB) which we can use as a block cache. No need for
> compaction of a raw data at all.
> Because we do not do any real compaction, we do not use CPU and IO (disk and
> network), we do not evict hot data from a block cache. The result: improved
> throughput and latency both write and read.
> See: https://github.com/facebook/rocksdb/wiki/FIFO-compaction-style
> h3. To enable FIFO compaction policy
> For table:
> {code}
> HTableDescriptor desc = new HTableDescriptor(tableName);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> For CF:
> {code}
> HColumnDescriptor desc = new HColumnDescriptor(family);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> Although region splitting is supported, for optimal performance it should be
> disabled, either by setting explicitly DisabledRegionSplitPolicy or by
> setting ConstantSizeRegionSplitPolicy and very large max region size. You
> will have to increase to a very large number store's blocking file number :
> *hbase.hstore.blockingStoreFiles* as well.
>
> h3. Limitations
> Do not use FIFO compaction if :
> * Table/CF has MIN_VERSION > 0
> * Table/CF has TTL = FOREVER (HColumnDescriptor.DEFAULT_TTL)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15061) Refactor StoreFileScanner creation to builder pattern
[
https://issues.apache.org/jira/browse/HBASE-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15075836#comment-15075836
]
Lars Hofhansl commented on HBASE-15061:
---
This is in a semi-hot code path (done for every Get request), so we should be
careful with creating more objects. (Probably OK, since we're seeking the
scanners later anyway, which is much more expensive; still something to
consider)
> Refactor StoreFileScanner creation to builder pattern
> -
>
> Key: HBASE-15061
> URL: https://issues.apache.org/jira/browse/HBASE-15061
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: hbase-15061.patch, hbase-15061.v2.patch
>
>
> There are several falvors of calls that creates a list of StoreFileScanners,
> and new feature have been added to this recently. This patch converts the
> somewhat difficult to read (need to go to javadoc) call:
> {code}
> // which args are the most relevant to this?
> - List sfScanners =
> StoreFileScanner.getScannersForStoreFiles(sfs,
> -cacheMobBlocks, true, false, false, readPt);
> {code}
> into one that is more literate:
> {code}
> // ah, very clearly we are using defaults except for the caching settings
> + List sfScanners = new
> StoreFileScanner.ListBuilder(sfs, readPt)
> + .withCacheBlocks(cacheMobBlocks).build();
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15059) Allow 0.94 to compile against Hadoop 2.7.x
[ https://issues.apache.org/jira/browse/HBASE-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15076053#comment-15076053 ] Lars Hofhansl commented on HBASE-15059: --- [~stack] you OK with committing V2? > Allow 0.94 to compile against Hadoop 2.7.x > -- > > Key: HBASE-15059 > URL: https://issues.apache.org/jira/browse/HBASE-15059 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Fix For: 0.94.28 > > Attachments: 15059-v2.txt, 15059.txt > > > Currently HBase 0.94 cannot be compiled against Hadoop 2.7. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15059) Allow 0.94 to compile against Hadoop 2.7.x
[ https://issues.apache.org/jira/browse/HBASE-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15076751#comment-15076751 ] Lars Hofhansl commented on HBASE-15059: --- Ping... :) Good to commit V2? > Allow 0.94 to compile against Hadoop 2.7.x > -- > > Key: HBASE-15059 > URL: https://issues.apache.org/jira/browse/HBASE-15059 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Fix For: 0.94.28 > > Attachments: 15059-v2.txt, 15059.txt > > > Currently HBase 0.94 cannot be compiled against Hadoop 2.7. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-3484) Replace memstore's ConcurrentSkipListMap with our own implementation
[ https://issues.apache.org/jira/browse/HBASE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15080698#comment-15080698 ] Lars Hofhansl commented on HBASE-3484: -- bq. CSLM optimizes for Comparable keys, so if you specify a custom comparator, then it has to wrap every key you insert with a wrapper object. Specializing CSLM for our purposes would easily save 64 bytes per entry on this. At least with JDK 8 this is not the case. CSLM works without extra objects with or without a custom comparator. (The Harmony class linked above has that problem, though.) > Replace memstore's ConcurrentSkipListMap with our own implementation > > > Key: HBASE-3484 > URL: https://issues.apache.org/jira/browse/HBASE-3484 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 0.92.0 >Reporter: Todd Lipcon >Priority: Critical > Attachments: WIP_HBASE-3484.patch, hierarchical-map.txt, > memstore_drag.png > > > By copy-pasting ConcurrentSkipListMap into HBase we can make two improvements > to it for our use case in MemStore: > - add an iterator.replace() method which should allow us to do upsert much > more cheaply > - implement a Set directly without having to do Map to > save one reference per entry > It turns out CSLM is in public domain from its development as part of JSR > 166, so we should be OK with licenses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14221) Reduce the number of time row comparison is done in a Scan
[
https://issues.apache.org/jira/browse/HBASE-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15080742#comment-15080742
]
Lars Hofhansl commented on HBASE-14221:
---
I'm all for a smaller version of the patch.
Also from HBASE-9969, we should explore the KeyValueScannerPriorityQueue that
[~mcorgan] added (not LoserTree)
> Reduce the number of time row comparison is done in a Scan
> --
>
> Key: HBASE-14221
> URL: https://issues.apache.org/jira/browse/HBASE-14221
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 14221-0.98-takeALook.txt, HBASE-14221.patch,
> HBASE-14221_1.patch, HBASE-14221_1.patch, HBASE-14221_6.patch,
> withmatchingRowspatch.png, withoutmatchingRowspatch.png
>
>
> When we tried to do some profiling with the PE tool found this.
> Currently we do row comparisons in 3 places in a simple Scan case.
> 1) ScanQueryMatcher
> {code}
>int ret = this.rowComparator.compareRows(curCell, cell);
> if (!this.isReversed) {
> if (ret <= -1) {
> return MatchCode.DONE;
> } else if (ret >= 1) {
> // could optimize this, if necessary?
> // Could also be called SEEK_TO_CURRENT_ROW, but this
> // should be rare/never happens.
> return MatchCode.SEEK_NEXT_ROW;
> }
> } else {
> if (ret <= -1) {
> return MatchCode.SEEK_NEXT_ROW;
> } else if (ret >= 1) {
> return MatchCode.DONE;
> }
> }
> {code}
> 2) In StoreScanner next() while starting to scan the row
> {code}
> if (!scannerContext.hasAnyLimit(LimitScope.BETWEEN_CELLS) ||
> matcher.curCell == null ||
> isNewRow || !CellUtil.matchingRow(peeked, matcher.curCell)) {
> this.countPerRow = 0;
> matcher.setToNewRow(peeked);
> }
> {code}
> Particularly to see if we are in a new row.
> 3) In HRegion
> {code}
> scannerContext.setKeepProgress(true);
> heap.next(results, scannerContext);
> scannerContext.setKeepProgress(tmpKeepProgress);
> nextKv = heap.peek();
> moreCellsInRow = moreCellsInRow(nextKv, currentRowCell);
> {code}
> Here again there are cases where we need to careful for a MultiCF case. Was
> trying to solve this for the MultiCF case but is having lot of cases to
> solve. But atleast for a single CF case I think these comparison can be
> reduced.
> So for a single CF case in the SQM we are able to find if we have crossed a
> row using the code pasted above in SQM. That comparison is definitely needed.
> Now in case of a single CF the HRegion is going to have only one element in
> the heap and so the 3rd comparison can surely be avoided if the
> StoreScanner.next() was over due to MatchCode.DONE caused by SQM.
> Coming to the 2nd compareRows that we do in StoreScanner. next() - even that
> can be avoided if we know that the previous next() call was over due to a new
> row. Doing all this I found that the compareRows in the profiler which was
> 19% got reduced to 13%. Initially we can solve for single CF case which can
> be extended to MultiCF cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15059) Allow 0.94 to compile against Hadoop 2.7.x
[ https://issues.apache.org/jira/browse/HBASE-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15081976#comment-15081976 ] Lars Hofhansl commented on HBASE-15059: --- I think I'd rather document the process (over recompiling PB every time, since that requires the correct version of protoc on the machine as you say, even when it is not needed). At least it is possible to build against Hadoop 7.x Another option is to not override the protobuf version to pull in, but require an explicit -Dprotobuf.version=2.5.0 (could even add a comment in pom.xml to that extend). That would make the failure explicit at runtime. Folks would then look into what failed and find the instructions here, which would include to call build-proto.sh. Trying not to overcomplicate things. :) Thoughts? > Allow 0.94 to compile against Hadoop 2.7.x > -- > > Key: HBASE-15059 > URL: https://issues.apache.org/jira/browse/HBASE-15059 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.94.28 > > Attachments: 15059-v2.txt, 15059.txt > > > Currently HBase 0.94 cannot be compiled against Hadoop 2.7. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15059) Allow 0.94 to compile against Hadoop 2.7.x
[ https://issues.apache.org/jira/browse/HBASE-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15082202#comment-15082202 ] Lars Hofhansl commented on HBASE-15059: --- bq. Where you going to doc how to make it work on 2.7/jdk8? In the release message for the release? That should be enough I'd say. Was thinking 3 places: # comment in the pom (where it defines the 2.7 profile), can have a prominent comment there, that pb classes need to be recompiled and how to do that # in the release notes here # in the release message for the release > Allow 0.94 to compile against Hadoop 2.7.x > -- > > Key: HBASE-15059 > URL: https://issues.apache.org/jira/browse/HBASE-15059 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.94.28 > > Attachments: 15059-v2.txt, 15059.txt > > > Currently HBase 0.94 cannot be compiled against Hadoop 2.7. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14468) Compaction improvements: FIFO compaction policy
[
https://issues.apache.org/jira/browse/HBASE-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15082378#comment-15082378
]
Lars Hofhansl commented on HBASE-14468:
---
Yeah... Will commit today or early tomorrow.
> Compaction improvements: FIFO compaction policy
> ---
>
> Key: HBASE-14468
> URL: https://issues.apache.org/jira/browse/HBASE-14468
> Project: HBase
> Issue Type: Improvement
> Components: Compaction, Performance
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14468-0.98.txt, HBASE-14468-v1.patch,
> HBASE-14468-v10.patch, HBASE-14468-v2.patch, HBASE-14468-v3.patch,
> HBASE-14468-v4.patch, HBASE-14468-v5.patch, HBASE-14468-v6.patch,
> HBASE-14468-v7.patch, HBASE-14468-v8.patch, HBASE-14468-v9.patch
>
>
> h2. FIFO Compaction
> h3. Introduction
> FIFO compaction policy selects only files which have all cells expired. The
> column family MUST have non-default TTL.
> Essentially, FIFO compactor does only one job: collects expired store files.
> These are some applications which could benefit the most:
> # Use it for very high volume raw data which has low TTL and which is the
> source of another data (after additional processing). Example: Raw
> time-series vs. time-based rollup aggregates and compacted time-series. We
> collect raw time-series and store them into CF with FIFO compaction policy,
> periodically we run task which creates rollup aggregates and compacts
> time-series, the original raw data can be discarded after that.
> # Use it for data which can be kept entirely in a a block cache (RAM/SSD).
> Say we have local SSD (1TB) which we can use as a block cache. No need for
> compaction of a raw data at all.
> Because we do not do any real compaction, we do not use CPU and IO (disk and
> network), we do not evict hot data from a block cache. The result: improved
> throughput and latency both write and read.
> See: https://github.com/facebook/rocksdb/wiki/FIFO-compaction-style
> h3. To enable FIFO compaction policy
> For table:
> {code}
> HTableDescriptor desc = new HTableDescriptor(tableName);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> For CF:
> {code}
> HColumnDescriptor desc = new HColumnDescriptor(family);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> Although region splitting is supported, for optimal performance it should be
> disabled, either by setting explicitly DisabledRegionSplitPolicy or by
> setting ConstantSizeRegionSplitPolicy and very large max region size. You
> will have to increase to a very large number store's blocking file number :
> *hbase.hstore.blockingStoreFiles* as well.
>
> h3. Limitations
> Do not use FIFO compaction if :
> * Table/CF has MIN_VERSION > 0
> * Table/CF has TTL = FOREVER (HColumnDescriptor.DEFAULT_TTL)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-15059) Allow 0.94 to compile against Hadoop 2.7.x
[ https://issues.apache.org/jira/browse/HBASE-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-15059. --- Resolution: Fixed Hadoop Flags: Reviewed Pushed to 0.94 only (added a comment into the pom about the extra build steps) > Allow 0.94 to compile against Hadoop 2.7.x > -- > > Key: HBASE-15059 > URL: https://issues.apache.org/jira/browse/HBASE-15059 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.94.28 > > Attachments: 15059-v2.txt, 15059.txt > > > Currently HBase 0.94 cannot be compiled against Hadoop 2.7. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15059) Allow 0.94 to compile against Hadoop 2.7.x
[ https://issues.apache.org/jira/browse/HBASE-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-15059: -- Release Note: This allow HBase to compile against Hadoop 2.7.1. TimeStampingFileContext.java had to be removed! If your code relied on this class to exist, do NOT upgrade to HBase 0.94.28. In order to work correctly with Hadoop 2.x+ the Java classes generated for protobuf need to generated with protoc 2.5.0 before HBase is compiled. This can be done with the included build-proto.sh script. Example: $ protoc --version check that the output indicates 2.5.0 $ dev-support/build-proto.sh $ mvn -Dhadoop.profile=2.7 ... was: This allow HBase to compile against Hadoop 2.7.1. TimeStampingFileContext.java had to be removed! If your code relied on this class to exist, do NOT upgrade to this version to HBase 0.94.28. In order to work correctly with Hadoop 2.x+ the Java classes generated for protobuf need to generated with protoc 2.5.0 before HBase is compiled. This can be done with the included build-proto.sh script. Example: $ protoc --version check that the output indicates 2.5.0 $ dev-support/build-proto.sh $ mvn -Dhadoop.profile=2.7 ... > Allow 0.94 to compile against Hadoop 2.7.x > -- > > Key: HBASE-15059 > URL: https://issues.apache.org/jira/browse/HBASE-15059 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.94.28 > > Attachments: 15059-v2.txt, 15059.txt > > > Currently HBase 0.94 cannot be compiled against Hadoop 2.7. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15059) Allow 0.94 to compile against Hadoop 2.7.x
[ https://issues.apache.org/jira/browse/HBASE-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-15059: -- Release Note: This allow HBase to compile against Hadoop 2.7.1. TimeStampingFileContext.java had to be removed! If your code relied on this class to exist, do NOT upgrade to this version to HBase 0.94.28. In order to work correctly with Hadoop 2.x+ the Java classes generated for protobuf need to generated with protoc 2.5.0 before HBase is compiled. This can be done with the included build-proto.sh script. Example: $ protoc --version check that the output indicates 2.5.0 $ dev-support/build-proto.sh $ mvn -Dhadoop.profile=2.7 ... > Allow 0.94 to compile against Hadoop 2.7.x > -- > > Key: HBASE-15059 > URL: https://issues.apache.org/jira/browse/HBASE-15059 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.94.28 > > Attachments: 15059-v2.txt, 15059.txt > > > Currently HBase 0.94 cannot be compiled against Hadoop 2.7. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-15054) Allow 0.94 to compile with JDK8
[ https://issues.apache.org/jira/browse/HBASE-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-15054. --- Resolution: Fixed Hadoop Flags: Reviewed > Allow 0.94 to compile with JDK8 > --- > > Key: HBASE-15054 > URL: https://issues.apache.org/jira/browse/HBASE-15054 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.94.28 > > Attachments: 15054.txt > > > Fix the following two problems: > # PoolMap > # InputSampler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15059) Allow 0.94 to compile against Hadoop 2.7.x
[ https://issues.apache.org/jira/browse/HBASE-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-15059: -- Release Note: This allows HBase to compile against Hadoop 2.7.1. TimeStampingFileContext.java had to be removed! If your code relied on this class to exist, do NOT upgrade to HBase 0.94.28. In order to work correctly with Hadoop 2.x+ the Java classes generated for protobuf need to generated with protoc 2.5.0 before HBase is compiled. This can be done with the included build-proto.sh script. Example: $ protoc --version check that the output indicates 2.5.0 $ dev-support/build-proto.sh $ mvn -Dhadoop.profile=2.7 ... was: This allow HBase to compile against Hadoop 2.7.1. TimeStampingFileContext.java had to be removed! If your code relied on this class to exist, do NOT upgrade to HBase 0.94.28. In order to work correctly with Hadoop 2.x+ the Java classes generated for protobuf need to generated with protoc 2.5.0 before HBase is compiled. This can be done with the included build-proto.sh script. Example: $ protoc --version check that the output indicates 2.5.0 $ dev-support/build-proto.sh $ mvn -Dhadoop.profile=2.7 ... > Allow 0.94 to compile against Hadoop 2.7.x > -- > > Key: HBASE-15059 > URL: https://issues.apache.org/jira/browse/HBASE-15059 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.94.28 > > Attachments: 15059-v2.txt, 15059.txt > > > Currently HBase 0.94 cannot be compiled against Hadoop 2.7. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-14213: -- Fix Version/s: (was: 0.94.28) > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-14213. --- Resolution: Won't Fix 0.94 is EOL'd. We're doing final release, no more versions after that. > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14468) Compaction improvements: FIFO compaction policy
[
https://issues.apache.org/jira/browse/HBASE-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15082464#comment-15082464
]
Lars Hofhansl commented on HBASE-14468:
---
Actually there's a bug in HMaster.checkCompactionPolicy(...):
{code}
+for (HColumnDescriptor hcd : htd.getColumnFamilies()) {
+ String compactionPolicy =
+ hcd.getConfigurationValue(DefaultStoreEngine.DEFAULT_COMPACTION_POLIC
Y_CLASS_KEY);
+ if (compactionPolicy == null) {
+compactionPolicy = className;
+ }
+ if (className.equals(FIFOCompactionPolicy.class.getName()) == false) {
<---
+continue;
{code}
The indicated line should be:
{code}
+ if (compactionPolicy.equals(FIFOCompactionPolicy.class.getName()) ==
false) {
{code}
No? (this is wrong in all branches)
> Compaction improvements: FIFO compaction policy
> ---
>
> Key: HBASE-14468
> URL: https://issues.apache.org/jira/browse/HBASE-14468
> Project: HBase
> Issue Type: Improvement
> Components: Compaction, Performance
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14468-0.98.txt, HBASE-14468-v1.patch,
> HBASE-14468-v10.patch, HBASE-14468-v2.patch, HBASE-14468-v3.patch,
> HBASE-14468-v4.patch, HBASE-14468-v5.patch, HBASE-14468-v6.patch,
> HBASE-14468-v7.patch, HBASE-14468-v8.patch, HBASE-14468-v9.patch
>
>
> h2. FIFO Compaction
> h3. Introduction
> FIFO compaction policy selects only files which have all cells expired. The
> column family MUST have non-default TTL.
> Essentially, FIFO compactor does only one job: collects expired store files.
> These are some applications which could benefit the most:
> # Use it for very high volume raw data which has low TTL and which is the
> source of another data (after additional processing). Example: Raw
> time-series vs. time-based rollup aggregates and compacted time-series. We
> collect raw time-series and store them into CF with FIFO compaction policy,
> periodically we run task which creates rollup aggregates and compacts
> time-series, the original raw data can be discarded after that.
> # Use it for data which can be kept entirely in a a block cache (RAM/SSD).
> Say we have local SSD (1TB) which we can use as a block cache. No need for
> compaction of a raw data at all.
> Because we do not do any real compaction, we do not use CPU and IO (disk and
> network), we do not evict hot data from a block cache. The result: improved
> throughput and latency both write and read.
> See: https://github.com/facebook/rocksdb/wiki/FIFO-compaction-style
> h3. To enable FIFO compaction policy
> For table:
> {code}
> HTableDescriptor desc = new HTableDescriptor(tableName);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> For CF:
> {code}
> HColumnDescriptor desc = new HColumnDescriptor(family);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> Although region splitting is supported, for optimal performance it should be
> disabled, either by setting explicitly DisabledRegionSplitPolicy or by
> setting ConstantSizeRegionSplitPolicy and very large max region size. You
> will have to increase to a very large number store's blocking file number :
> *hbase.hstore.blockingStoreFiles* as well.
>
> h3. Limitations
> Do not use FIFO compaction if :
> * Table/CF has MIN_VERSION > 0
> * Table/CF has TTL = FOREVER (HColumnDescriptor.DEFAULT_TTL)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14468) Compaction improvements: FIFO compaction policy
[
https://issues.apache.org/jira/browse/HBASE-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15082479#comment-15082479
]
Lars Hofhansl commented on HBASE-14468:
---
And (cosmetic) majorCompactionPeriod and splitPolicyClassName are not used
anywhere.
> Compaction improvements: FIFO compaction policy
> ---
>
> Key: HBASE-14468
> URL: https://issues.apache.org/jira/browse/HBASE-14468
> Project: HBase
> Issue Type: Improvement
> Components: Compaction, Performance
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14468-0.98.txt, HBASE-14468-v1.patch,
> HBASE-14468-v10.patch, HBASE-14468-v2.patch, HBASE-14468-v3.patch,
> HBASE-14468-v4.patch, HBASE-14468-v5.patch, HBASE-14468-v6.patch,
> HBASE-14468-v7.patch, HBASE-14468-v8.patch, HBASE-14468-v9.patch
>
>
> h2. FIFO Compaction
> h3. Introduction
> FIFO compaction policy selects only files which have all cells expired. The
> column family MUST have non-default TTL.
> Essentially, FIFO compactor does only one job: collects expired store files.
> These are some applications which could benefit the most:
> # Use it for very high volume raw data which has low TTL and which is the
> source of another data (after additional processing). Example: Raw
> time-series vs. time-based rollup aggregates and compacted time-series. We
> collect raw time-series and store them into CF with FIFO compaction policy,
> periodically we run task which creates rollup aggregates and compacts
> time-series, the original raw data can be discarded after that.
> # Use it for data which can be kept entirely in a a block cache (RAM/SSD).
> Say we have local SSD (1TB) which we can use as a block cache. No need for
> compaction of a raw data at all.
> Because we do not do any real compaction, we do not use CPU and IO (disk and
> network), we do not evict hot data from a block cache. The result: improved
> throughput and latency both write and read.
> See: https://github.com/facebook/rocksdb/wiki/FIFO-compaction-style
> h3. To enable FIFO compaction policy
> For table:
> {code}
> HTableDescriptor desc = new HTableDescriptor(tableName);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> For CF:
> {code}
> HColumnDescriptor desc = new HColumnDescriptor(family);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> Although region splitting is supported, for optimal performance it should be
> disabled, either by setting explicitly DisabledRegionSplitPolicy or by
> setting ConstantSizeRegionSplitPolicy and very large max region size. You
> will have to increase to a very large number store's blocking file number :
> *hbase.hstore.blockingStoreFiles* as well.
>
> h3. Limitations
> Do not use FIFO compaction if :
> * Table/CF has MIN_VERSION > 0
> * Table/CF has TTL = FOREVER (HColumnDescriptor.DEFAULT_TTL)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14468) Compaction improvements: FIFO compaction policy
[
https://issues.apache.org/jira/browse/HBASE-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15082514#comment-15082514
]
Lars Hofhansl commented on HBASE-14468:
---
Lastly, does this require HBASE-14467 as well?
> Compaction improvements: FIFO compaction policy
> ---
>
> Key: HBASE-14468
> URL: https://issues.apache.org/jira/browse/HBASE-14468
> Project: HBase
> Issue Type: Improvement
> Components: Compaction, Performance
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14468-0.98.txt, HBASE-14468-v1.patch,
> HBASE-14468-v10.patch, HBASE-14468-v2.patch, HBASE-14468-v3.patch,
> HBASE-14468-v4.patch, HBASE-14468-v5.patch, HBASE-14468-v6.patch,
> HBASE-14468-v7.patch, HBASE-14468-v8.patch, HBASE-14468-v9.patch
>
>
> h2. FIFO Compaction
> h3. Introduction
> FIFO compaction policy selects only files which have all cells expired. The
> column family MUST have non-default TTL.
> Essentially, FIFO compactor does only one job: collects expired store files.
> These are some applications which could benefit the most:
> # Use it for very high volume raw data which has low TTL and which is the
> source of another data (after additional processing). Example: Raw
> time-series vs. time-based rollup aggregates and compacted time-series. We
> collect raw time-series and store them into CF with FIFO compaction policy,
> periodically we run task which creates rollup aggregates and compacts
> time-series, the original raw data can be discarded after that.
> # Use it for data which can be kept entirely in a a block cache (RAM/SSD).
> Say we have local SSD (1TB) which we can use as a block cache. No need for
> compaction of a raw data at all.
> Because we do not do any real compaction, we do not use CPU and IO (disk and
> network), we do not evict hot data from a block cache. The result: improved
> throughput and latency both write and read.
> See: https://github.com/facebook/rocksdb/wiki/FIFO-compaction-style
> h3. To enable FIFO compaction policy
> For table:
> {code}
> HTableDescriptor desc = new HTableDescriptor(tableName);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> For CF:
> {code}
> HColumnDescriptor desc = new HColumnDescriptor(family);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> Although region splitting is supported, for optimal performance it should be
> disabled, either by setting explicitly DisabledRegionSplitPolicy or by
> setting ConstantSizeRegionSplitPolicy and very large max region size. You
> will have to increase to a very large number store's blocking file number :
> *hbase.hstore.blockingStoreFiles* as well.
>
> h3. Limitations
> Do not use FIFO compaction if :
> * Table/CF has MIN_VERSION > 0
> * Table/CF has TTL = FOREVER (HColumnDescriptor.DEFAULT_TTL)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14468) Compaction improvements: FIFO compaction policy
[
https://issues.apache.org/jira/browse/HBASE-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15082522#comment-15082522
]
Lars Hofhansl commented on HBASE-14468:
---
One more: Should we recommend increasing hbase.hstore.compactionThreshold as
well (default it 3)
> Compaction improvements: FIFO compaction policy
> ---
>
> Key: HBASE-14468
> URL: https://issues.apache.org/jira/browse/HBASE-14468
> Project: HBase
> Issue Type: Improvement
> Components: Compaction, Performance
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14468-0.98.txt, HBASE-14468-v1.patch,
> HBASE-14468-v10.patch, HBASE-14468-v2.patch, HBASE-14468-v3.patch,
> HBASE-14468-v4.patch, HBASE-14468-v5.patch, HBASE-14468-v6.patch,
> HBASE-14468-v7.patch, HBASE-14468-v8.patch, HBASE-14468-v9.patch
>
>
> h2. FIFO Compaction
> h3. Introduction
> FIFO compaction policy selects only files which have all cells expired. The
> column family MUST have non-default TTL.
> Essentially, FIFO compactor does only one job: collects expired store files.
> These are some applications which could benefit the most:
> # Use it for very high volume raw data which has low TTL and which is the
> source of another data (after additional processing). Example: Raw
> time-series vs. time-based rollup aggregates and compacted time-series. We
> collect raw time-series and store them into CF with FIFO compaction policy,
> periodically we run task which creates rollup aggregates and compacts
> time-series, the original raw data can be discarded after that.
> # Use it for data which can be kept entirely in a a block cache (RAM/SSD).
> Say we have local SSD (1TB) which we can use as a block cache. No need for
> compaction of a raw data at all.
> Because we do not do any real compaction, we do not use CPU and IO (disk and
> network), we do not evict hot data from a block cache. The result: improved
> throughput and latency both write and read.
> See: https://github.com/facebook/rocksdb/wiki/FIFO-compaction-style
> h3. To enable FIFO compaction policy
> For table:
> {code}
> HTableDescriptor desc = new HTableDescriptor(tableName);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> For CF:
> {code}
> HColumnDescriptor desc = new HColumnDescriptor(family);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> Although region splitting is supported, for optimal performance it should be
> disabled, either by setting explicitly DisabledRegionSplitPolicy or by
> setting ConstantSizeRegionSplitPolicy and very large max region size. You
> will have to increase to a very large number store's blocking file number :
> *hbase.hstore.blockingStoreFiles* as well.
>
> h3. Limitations
> Do not use FIFO compaction if :
> * Table/CF has MIN_VERSION > 0
> * Table/CF has TTL = FOREVER (HColumnDescriptor.DEFAULT_TTL)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14468) Compaction improvements: FIFO compaction policy
[
https://issues.apache.org/jira/browse/HBASE-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15082532#comment-15082532
]
Lars Hofhansl commented on HBASE-14468:
---
Example shell command:
{code}
create 'x',{NAME=>'y', TTL=>'30'}, {CONFIGURATION =>
{'hbase.hstore.defaultengine.compactionpolicy.class' =>
'org.apache.hadoop.hbase.regionserver.compactions.FIFOCompactionPolicy',
'hbase.hstore.blockingStoreFiles' => 1000}}
{code}
> Compaction improvements: FIFO compaction policy
> ---
>
> Key: HBASE-14468
> URL: https://issues.apache.org/jira/browse/HBASE-14468
> Project: HBase
> Issue Type: Improvement
> Components: Compaction, Performance
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14468-0.98.txt, HBASE-14468-v1.patch,
> HBASE-14468-v10.patch, HBASE-14468-v2.patch, HBASE-14468-v3.patch,
> HBASE-14468-v4.patch, HBASE-14468-v5.patch, HBASE-14468-v6.patch,
> HBASE-14468-v7.patch, HBASE-14468-v8.patch, HBASE-14468-v9.patch
>
>
> h2. FIFO Compaction
> h3. Introduction
> FIFO compaction policy selects only files which have all cells expired. The
> column family MUST have non-default TTL.
> Essentially, FIFO compactor does only one job: collects expired store files.
> These are some applications which could benefit the most:
> # Use it for very high volume raw data which has low TTL and which is the
> source of another data (after additional processing). Example: Raw
> time-series vs. time-based rollup aggregates and compacted time-series. We
> collect raw time-series and store them into CF with FIFO compaction policy,
> periodically we run task which creates rollup aggregates and compacts
> time-series, the original raw data can be discarded after that.
> # Use it for data which can be kept entirely in a a block cache (RAM/SSD).
> Say we have local SSD (1TB) which we can use as a block cache. No need for
> compaction of a raw data at all.
> Because we do not do any real compaction, we do not use CPU and IO (disk and
> network), we do not evict hot data from a block cache. The result: improved
> throughput and latency both write and read.
> See: https://github.com/facebook/rocksdb/wiki/FIFO-compaction-style
> h3. To enable FIFO compaction policy
> For table:
> {code}
> HTableDescriptor desc = new HTableDescriptor(tableName);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> For CF:
> {code}
> HColumnDescriptor desc = new HColumnDescriptor(family);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> Although region splitting is supported, for optimal performance it should be
> disabled, either by setting explicitly DisabledRegionSplitPolicy or by
> setting ConstantSizeRegionSplitPolicy and very large max region size. You
> will have to increase to a very large number store's blocking file number :
> *hbase.hstore.blockingStoreFiles* as well.
>
> h3. Limitations
> Do not use FIFO compaction if :
> * Table/CF has MIN_VERSION > 0
> * Table/CF has TTL = FOREVER (HColumnDescriptor.DEFAULT_TTL)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14468) Compaction improvements: FIFO compaction policy
[
https://issues.apache.org/jira/browse/HBASE-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-14468:
--
Attachment: 14468-0.98-v2.txt
V2 for 0.98.
* uses compactionPolicy correctly in checkCompactionPolicy
* gets rid of a bunch of trailing white space (I hate to the whitespace nazi :)
)
I did some basic testing on this, but nothing intensive.
> Compaction improvements: FIFO compaction policy
> ---
>
> Key: HBASE-14468
> URL: https://issues.apache.org/jira/browse/HBASE-14468
> Project: HBase
> Issue Type: Improvement
> Components: Compaction, Performance
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14468-0.98-v2.txt, 14468-0.98.txt, HBASE-14468-v1.patch,
> HBASE-14468-v10.patch, HBASE-14468-v2.patch, HBASE-14468-v3.patch,
> HBASE-14468-v4.patch, HBASE-14468-v5.patch, HBASE-14468-v6.patch,
> HBASE-14468-v7.patch, HBASE-14468-v8.patch, HBASE-14468-v9.patch
>
>
> h2. FIFO Compaction
> h3. Introduction
> FIFO compaction policy selects only files which have all cells expired. The
> column family MUST have non-default TTL.
> Essentially, FIFO compactor does only one job: collects expired store files.
> These are some applications which could benefit the most:
> # Use it for very high volume raw data which has low TTL and which is the
> source of another data (after additional processing). Example: Raw
> time-series vs. time-based rollup aggregates and compacted time-series. We
> collect raw time-series and store them into CF with FIFO compaction policy,
> periodically we run task which creates rollup aggregates and compacts
> time-series, the original raw data can be discarded after that.
> # Use it for data which can be kept entirely in a a block cache (RAM/SSD).
> Say we have local SSD (1TB) which we can use as a block cache. No need for
> compaction of a raw data at all.
> Because we do not do any real compaction, we do not use CPU and IO (disk and
> network), we do not evict hot data from a block cache. The result: improved
> throughput and latency both write and read.
> See: https://github.com/facebook/rocksdb/wiki/FIFO-compaction-style
> h3. To enable FIFO compaction policy
> For table:
> {code}
> HTableDescriptor desc = new HTableDescriptor(tableName);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> For CF:
> {code}
> HColumnDescriptor desc = new HColumnDescriptor(family);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> Although region splitting is supported, for optimal performance it should be
> disabled, either by setting explicitly DisabledRegionSplitPolicy or by
> setting ConstantSizeRegionSplitPolicy and very large max region size. You
> will have to increase to a very large number store's blocking file number :
> *hbase.hstore.blockingStoreFiles* as well.
>
> h3. Limitations
> Do not use FIFO compaction if :
> * Table/CF has MIN_VERSION > 0
> * Table/CF has TTL = FOREVER (HColumnDescriptor.DEFAULT_TTL)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14467) Compaction improvements: DefaultCompactor should not compact TTL-expired files
[ https://issues.apache.org/jira/browse/HBASE-14467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15082543#comment-15082543 ] Lars Hofhansl commented on HBASE-14467: --- This is "resolved" as "fixed", yet I see no commit for this. > Compaction improvements: DefaultCompactor should not compact TTL-expired files > -- > > Key: HBASE-14467 > URL: https://issues.apache.org/jira/browse/HBASE-14467 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-14467-v1.patch, HBASE-14467-v2.patch, > HBASE-14467-v3.patch > > > DefaultCompactor does not check if store file can be completely purged > because of TTL expiration. That is simple optimization which allows to > implement such "exotic"compaction policy as FIFO compaction > (https://github.com/facebook/rocksdb/wiki/FIFO-compaction-style). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14468) Compaction improvements: FIFO compaction policy
[
https://issues.apache.org/jira/browse/HBASE-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15082566#comment-15082566
]
Lars Hofhansl commented on HBASE-14468:
---
Sorry for the spam here. I tested the following scenario:
* compaction threshold set to 3
* TTL 30s
* flushed 5 files to disk within 30s.
Shouldn't I expect that after a while all files should be removed? I find them
still hanging around after 1h.
> Compaction improvements: FIFO compaction policy
> ---
>
> Key: HBASE-14468
> URL: https://issues.apache.org/jira/browse/HBASE-14468
> Project: HBase
> Issue Type: Improvement
> Components: Compaction, Performance
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14468-0.98-v2.txt, 14468-0.98.txt, HBASE-14468-v1.patch,
> HBASE-14468-v10.patch, HBASE-14468-v2.patch, HBASE-14468-v3.patch,
> HBASE-14468-v4.patch, HBASE-14468-v5.patch, HBASE-14468-v6.patch,
> HBASE-14468-v7.patch, HBASE-14468-v8.patch, HBASE-14468-v9.patch
>
>
> h2. FIFO Compaction
> h3. Introduction
> FIFO compaction policy selects only files which have all cells expired. The
> column family MUST have non-default TTL.
> Essentially, FIFO compactor does only one job: collects expired store files.
> These are some applications which could benefit the most:
> # Use it for very high volume raw data which has low TTL and which is the
> source of another data (after additional processing). Example: Raw
> time-series vs. time-based rollup aggregates and compacted time-series. We
> collect raw time-series and store them into CF with FIFO compaction policy,
> periodically we run task which creates rollup aggregates and compacts
> time-series, the original raw data can be discarded after that.
> # Use it for data which can be kept entirely in a a block cache (RAM/SSD).
> Say we have local SSD (1TB) which we can use as a block cache. No need for
> compaction of a raw data at all.
> Because we do not do any real compaction, we do not use CPU and IO (disk and
> network), we do not evict hot data from a block cache. The result: improved
> throughput and latency both write and read.
> See: https://github.com/facebook/rocksdb/wiki/FIFO-compaction-style
> h3. To enable FIFO compaction policy
> For table:
> {code}
> HTableDescriptor desc = new HTableDescriptor(tableName);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> For CF:
> {code}
> HColumnDescriptor desc = new HColumnDescriptor(family);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> Although region splitting is supported, for optimal performance it should be
> disabled, either by setting explicitly DisabledRegionSplitPolicy or by
> setting ConstantSizeRegionSplitPolicy and very large max region size. You
> will have to increase to a very large number store's blocking file number :
> *hbase.hstore.blockingStoreFiles* as well.
>
> h3. Limitations
> Do not use FIFO compaction if :
> * Table/CF has MIN_VERSION > 0
> * Table/CF has TTL = FOREVER (HColumnDescriptor.DEFAULT_TTL)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14221) Reduce the number of time row comparison is done in a Scan
[
https://issues.apache.org/jira/browse/HBASE-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15083466#comment-15083466
]
Lars Hofhansl commented on HBASE-14221:
---
+1
(please use "lhofhansl" as alias, not "larsh" :) )
> Reduce the number of time row comparison is done in a Scan
> --
>
> Key: HBASE-14221
> URL: https://issues.apache.org/jira/browse/HBASE-14221
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 14221-0.98-takeALook.txt, HBASE-14221.patch,
> HBASE-14221_1.patch, HBASE-14221_1.patch, HBASE-14221_6.patch,
> HBASE-14221_9.patch, withmatchingRowspatch.png, withoutmatchingRowspatch.png
>
>
> When we tried to do some profiling with the PE tool found this.
> Currently we do row comparisons in 3 places in a simple Scan case.
> 1) ScanQueryMatcher
> {code}
>int ret = this.rowComparator.compareRows(curCell, cell);
> if (!this.isReversed) {
> if (ret <= -1) {
> return MatchCode.DONE;
> } else if (ret >= 1) {
> // could optimize this, if necessary?
> // Could also be called SEEK_TO_CURRENT_ROW, but this
> // should be rare/never happens.
> return MatchCode.SEEK_NEXT_ROW;
> }
> } else {
> if (ret <= -1) {
> return MatchCode.SEEK_NEXT_ROW;
> } else if (ret >= 1) {
> return MatchCode.DONE;
> }
> }
> {code}
> 2) In StoreScanner next() while starting to scan the row
> {code}
> if (!scannerContext.hasAnyLimit(LimitScope.BETWEEN_CELLS) ||
> matcher.curCell == null ||
> isNewRow || !CellUtil.matchingRow(peeked, matcher.curCell)) {
> this.countPerRow = 0;
> matcher.setToNewRow(peeked);
> }
> {code}
> Particularly to see if we are in a new row.
> 3) In HRegion
> {code}
> scannerContext.setKeepProgress(true);
> heap.next(results, scannerContext);
> scannerContext.setKeepProgress(tmpKeepProgress);
> nextKv = heap.peek();
> moreCellsInRow = moreCellsInRow(nextKv, currentRowCell);
> {code}
> Here again there are cases where we need to careful for a MultiCF case. Was
> trying to solve this for the MultiCF case but is having lot of cases to
> solve. But atleast for a single CF case I think these comparison can be
> reduced.
> So for a single CF case in the SQM we are able to find if we have crossed a
> row using the code pasted above in SQM. That comparison is definitely needed.
> Now in case of a single CF the HRegion is going to have only one element in
> the heap and so the 3rd comparison can surely be avoided if the
> StoreScanner.next() was over due to MatchCode.DONE caused by SQM.
> Coming to the 2nd compareRows that we do in StoreScanner. next() - even that
> can be avoided if we know that the previous next() call was over due to a new
> row. Doing all this I found that the compareRows in the profiler which was
> 19% got reduced to 13%. Initially we can solve for single CF case which can
> be extended to MultiCF cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14468) Compaction improvements: FIFO compaction policy
[
https://issues.apache.org/jira/browse/HBASE-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15084059#comment-15084059
]
Lars Hofhansl commented on HBASE-14468:
---
That's different. I see that as I flush at the time of flushing we do
compaction checks and remove the old files.
But if I write a bunch in 30s (I flush manually), and then stop all activity
the files are not purged.
With longer TTLs of a few days or weeks that may significant.
I tested this is the shell with:
{code}
put 'x', 'r1', 'y:1', 1; flush 'x'
{code}
And just changing 'r1' to 'r2', 'r3', etc, each time. That way I created 6
store files in 30s, and they never get collected, even though their TTL clearly
expires. (compaction threshold is still 3). With the default collector they do
get collected after a minute or so.
Is there something else besides this patch I need that is 1.2+ but not in 0.98?
> Compaction improvements: FIFO compaction policy
> ---
>
> Key: HBASE-14468
> URL: https://issues.apache.org/jira/browse/HBASE-14468
> Project: HBase
> Issue Type: Improvement
> Components: Compaction, Performance
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14468-0.98-v2.txt, 14468-0.98.txt, HBASE-14468-v1.patch,
> HBASE-14468-v10.patch, HBASE-14468-v2.patch, HBASE-14468-v3.patch,
> HBASE-14468-v4.patch, HBASE-14468-v5.patch, HBASE-14468-v6.patch,
> HBASE-14468-v7.patch, HBASE-14468-v8.patch, HBASE-14468-v9.patch,
> HBASE-14468.add.patch
>
>
> h2. FIFO Compaction
> h3. Introduction
> FIFO compaction policy selects only files which have all cells expired. The
> column family MUST have non-default TTL.
> Essentially, FIFO compactor does only one job: collects expired store files.
> These are some applications which could benefit the most:
> # Use it for very high volume raw data which has low TTL and which is the
> source of another data (after additional processing). Example: Raw
> time-series vs. time-based rollup aggregates and compacted time-series. We
> collect raw time-series and store them into CF with FIFO compaction policy,
> periodically we run task which creates rollup aggregates and compacts
> time-series, the original raw data can be discarded after that.
> # Use it for data which can be kept entirely in a a block cache (RAM/SSD).
> Say we have local SSD (1TB) which we can use as a block cache. No need for
> compaction of a raw data at all.
> Because we do not do any real compaction, we do not use CPU and IO (disk and
> network), we do not evict hot data from a block cache. The result: improved
> throughput and latency both write and read.
> See: https://github.com/facebook/rocksdb/wiki/FIFO-compaction-style
> h3. To enable FIFO compaction policy
> For table:
> {code}
> HTableDescriptor desc = new HTableDescriptor(tableName);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> For CF:
> {code}
> HColumnDescriptor desc = new HColumnDescriptor(family);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> Although region splitting is supported, for optimal performance it should be
> disabled, either by setting explicitly DisabledRegionSplitPolicy or by
> setting ConstantSizeRegionSplitPolicy and very large max region size. You
> will have to increase to a very large number store's blocking file number :
> *hbase.hstore.blockingStoreFiles* as well.
>
> h3. Limitations
> Do not use FIFO compaction if :
> * Table/CF has MIN_VERSION > 0
> * Table/CF has TTL = FOREVER (HColumnDescriptor.DEFAULT_TTL)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14468) Compaction improvements: FIFO compaction policy
[
https://issues.apache.org/jira/browse/HBASE-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15084061#comment-15084061
]
Lars Hofhansl commented on HBASE-14468:
---
+1 on addendum.
> Compaction improvements: FIFO compaction policy
> ---
>
> Key: HBASE-14468
> URL: https://issues.apache.org/jira/browse/HBASE-14468
> Project: HBase
> Issue Type: Improvement
> Components: Compaction, Performance
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14468-0.98-v2.txt, 14468-0.98.txt, HBASE-14468-v1.patch,
> HBASE-14468-v10.patch, HBASE-14468-v2.patch, HBASE-14468-v3.patch,
> HBASE-14468-v4.patch, HBASE-14468-v5.patch, HBASE-14468-v6.patch,
> HBASE-14468-v7.patch, HBASE-14468-v8.patch, HBASE-14468-v9.patch,
> HBASE-14468.add.patch
>
>
> h2. FIFO Compaction
> h3. Introduction
> FIFO compaction policy selects only files which have all cells expired. The
> column family MUST have non-default TTL.
> Essentially, FIFO compactor does only one job: collects expired store files.
> These are some applications which could benefit the most:
> # Use it for very high volume raw data which has low TTL and which is the
> source of another data (after additional processing). Example: Raw
> time-series vs. time-based rollup aggregates and compacted time-series. We
> collect raw time-series and store them into CF with FIFO compaction policy,
> periodically we run task which creates rollup aggregates and compacts
> time-series, the original raw data can be discarded after that.
> # Use it for data which can be kept entirely in a a block cache (RAM/SSD).
> Say we have local SSD (1TB) which we can use as a block cache. No need for
> compaction of a raw data at all.
> Because we do not do any real compaction, we do not use CPU and IO (disk and
> network), we do not evict hot data from a block cache. The result: improved
> throughput and latency both write and read.
> See: https://github.com/facebook/rocksdb/wiki/FIFO-compaction-style
> h3. To enable FIFO compaction policy
> For table:
> {code}
> HTableDescriptor desc = new HTableDescriptor(tableName);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> For CF:
> {code}
> HColumnDescriptor desc = new HColumnDescriptor(family);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> Although region splitting is supported, for optimal performance it should be
> disabled, either by setting explicitly DisabledRegionSplitPolicy or by
> setting ConstantSizeRegionSplitPolicy and very large max region size. You
> will have to increase to a very large number store's blocking file number :
> *hbase.hstore.blockingStoreFiles* as well.
>
> h3. Limitations
> Do not use FIFO compaction if :
> * Table/CF has MIN_VERSION > 0
> * Table/CF has TTL = FOREVER (HColumnDescriptor.DEFAULT_TTL)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14468) Compaction improvements: FIFO compaction policy
[
https://issues.apache.org/jira/browse/HBASE-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15084059#comment-15084059
]
Lars Hofhansl edited comment on HBASE-14468 at 1/5/16 11:41 PM:
That's different. I see that as I flush at the time of flushing we do
compaction checks and remove the old files.
But if I write a bunch in 30s (I flush manually), and then stop all activity
the files are not purged.
With longer TTLs of a few days or weeks that may significant.
I tested this in the shell with:
{code}
put 'x', 'r1', 'y:1', 1; flush 'x'
{code}
And just changing 'r1' to 'r2', 'r3', etc, each time. That way I created 5
store files in 30s. Then I stop all activity, and the store files never get
collected, even though their TTL clearly expires. (compaction threshold is
still 3). With the default collector they do get collected after a minute or so.
Is there something else besides this patch I need that is 1.2+ but not in 0.98?
was (Author: lhofhansl):
That's different. I see that as I flush at the time of flushing we do
compaction checks and remove the old files.
But if I write a bunch in 30s (I flush manually), and then stop all activity
the files are not purged.
With longer TTLs of a few days or weeks that may significant.
I tested this is the shell with:
{code}
put 'x', 'r1', 'y:1', 1; flush 'x'
{code}
And just changing 'r1' to 'r2', 'r3', etc, each time. That way I created 6
store files in 30s, and they never get collected, even though their TTL clearly
expires. (compaction threshold is still 3). With the default collector they do
get collected after a minute or so.
Is there something else besides this patch I need that is 1.2+ but not in 0.98?
> Compaction improvements: FIFO compaction policy
> ---
>
> Key: HBASE-14468
> URL: https://issues.apache.org/jira/browse/HBASE-14468
> Project: HBase
> Issue Type: Improvement
> Components: Compaction, Performance
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14468-0.98-v2.txt, 14468-0.98.txt, HBASE-14468-v1.patch,
> HBASE-14468-v10.patch, HBASE-14468-v2.patch, HBASE-14468-v3.patch,
> HBASE-14468-v4.patch, HBASE-14468-v5.patch, HBASE-14468-v6.patch,
> HBASE-14468-v7.patch, HBASE-14468-v8.patch, HBASE-14468-v9.patch,
> HBASE-14468.add.patch
>
>
> h2. FIFO Compaction
> h3. Introduction
> FIFO compaction policy selects only files which have all cells expired. The
> column family MUST have non-default TTL.
> Essentially, FIFO compactor does only one job: collects expired store files.
> These are some applications which could benefit the most:
> # Use it for very high volume raw data which has low TTL and which is the
> source of another data (after additional processing). Example: Raw
> time-series vs. time-based rollup aggregates and compacted time-series. We
> collect raw time-series and store them into CF with FIFO compaction policy,
> periodically we run task which creates rollup aggregates and compacts
> time-series, the original raw data can be discarded after that.
> # Use it for data which can be kept entirely in a a block cache (RAM/SSD).
> Say we have local SSD (1TB) which we can use as a block cache. No need for
> compaction of a raw data at all.
> Because we do not do any real compaction, we do not use CPU and IO (disk and
> network), we do not evict hot data from a block cache. The result: improved
> throughput and latency both write and read.
> See: https://github.com/facebook/rocksdb/wiki/FIFO-compaction-style
> h3. To enable FIFO compaction policy
> For table:
> {code}
> HTableDescriptor desc = new HTableDescriptor(tableName);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> For CF:
> {code}
> HColumnDescriptor desc = new HColumnDescriptor(family);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> Although region splitting is supported, for optimal performance it should be
> disabled, either by setting explicitly DisabledRegionSplitPolicy or by
> setting ConstantSizeRegionSplitPolicy and very large max region size. You
> will have to increase to a very large number store's blocking file number :
> *hbase.hstore.blockingStoreFiles* as well.
>
> h3. Limitations
> Do not use FIFO compaction if :
> * Table/CF has MIN_VERSION > 0
> * Table/CF has TTL = FOREVER (HColumnDescriptor.DEFAULT_TTL)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14468) Compaction improvements: FIFO compaction policy
[
https://issues.apache.org/jira/browse/HBASE-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15084099#comment-15084099
]
Lars Hofhansl commented on HBASE-14468:
---
Pushed addendum to 1.2, 1.3, and 2.0
> Compaction improvements: FIFO compaction policy
> ---
>
> Key: HBASE-14468
> URL: https://issues.apache.org/jira/browse/HBASE-14468
> Project: HBase
> Issue Type: Improvement
> Components: Compaction, Performance
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14468-0.98-v2.txt, 14468-0.98.txt, HBASE-14468-v1.patch,
> HBASE-14468-v10.patch, HBASE-14468-v2.patch, HBASE-14468-v3.patch,
> HBASE-14468-v4.patch, HBASE-14468-v5.patch, HBASE-14468-v6.patch,
> HBASE-14468-v7.patch, HBASE-14468-v8.patch, HBASE-14468-v9.patch,
> HBASE-14468.add.patch
>
>
> h2. FIFO Compaction
> h3. Introduction
> FIFO compaction policy selects only files which have all cells expired. The
> column family MUST have non-default TTL.
> Essentially, FIFO compactor does only one job: collects expired store files.
> These are some applications which could benefit the most:
> # Use it for very high volume raw data which has low TTL and which is the
> source of another data (after additional processing). Example: Raw
> time-series vs. time-based rollup aggregates and compacted time-series. We
> collect raw time-series and store them into CF with FIFO compaction policy,
> periodically we run task which creates rollup aggregates and compacts
> time-series, the original raw data can be discarded after that.
> # Use it for data which can be kept entirely in a a block cache (RAM/SSD).
> Say we have local SSD (1TB) which we can use as a block cache. No need for
> compaction of a raw data at all.
> Because we do not do any real compaction, we do not use CPU and IO (disk and
> network), we do not evict hot data from a block cache. The result: improved
> throughput and latency both write and read.
> See: https://github.com/facebook/rocksdb/wiki/FIFO-compaction-style
> h3. To enable FIFO compaction policy
> For table:
> {code}
> HTableDescriptor desc = new HTableDescriptor(tableName);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> For CF:
> {code}
> HColumnDescriptor desc = new HColumnDescriptor(family);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> Although region splitting is supported, for optimal performance it should be
> disabled, either by setting explicitly DisabledRegionSplitPolicy or by
> setting ConstantSizeRegionSplitPolicy and very large max region size. You
> will have to increase to a very large number store's blocking file number :
> *hbase.hstore.blockingStoreFiles* as well.
>
> h3. Limitations
> Do not use FIFO compaction if :
> * Table/CF has MIN_VERSION > 0
> * Table/CF has TTL = FOREVER (HColumnDescriptor.DEFAULT_TTL)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15084980#comment-15084980 ] Lars Hofhansl commented on HBASE-14213: --- Thanks [~busbey]. Anything I can do to help? (I was ready to do the release today, when I saw this issue reopened) > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14468) Compaction improvements: FIFO compaction policy
[
https://issues.apache.org/jira/browse/HBASE-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15086059#comment-15086059
]
Lars Hofhansl commented on HBASE-14468:
---
bq. HBASE-10141. Not sure if it is in 0.98.
That's in 0.98 via HBASE-12144
[~vrodionov], any change to try the scenario I mentioned above?
I.e. (1) define a TTL (perhaps a few minutes), (2) do a bunch of flushed within
the TTL, (3) then stop all writes, (4) see if all prior store files are
eventually collected without any write activity.
I might get time to test a bit more in 0.98, until then I'm gonna hold off
committing to 0.98.
> Compaction improvements: FIFO compaction policy
> ---
>
> Key: HBASE-14468
> URL: https://issues.apache.org/jira/browse/HBASE-14468
> Project: HBase
> Issue Type: Improvement
> Components: Compaction, Performance
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14468-0.98-v2.txt, 14468-0.98.txt, HBASE-14468-v1.patch,
> HBASE-14468-v10.patch, HBASE-14468-v2.patch, HBASE-14468-v3.patch,
> HBASE-14468-v4.patch, HBASE-14468-v5.patch, HBASE-14468-v6.patch,
> HBASE-14468-v7.patch, HBASE-14468-v8.patch, HBASE-14468-v9.patch,
> HBASE-14468.add.patch
>
>
> h2. FIFO Compaction
> h3. Introduction
> FIFO compaction policy selects only files which have all cells expired. The
> column family MUST have non-default TTL.
> Essentially, FIFO compactor does only one job: collects expired store files.
> These are some applications which could benefit the most:
> # Use it for very high volume raw data which has low TTL and which is the
> source of another data (after additional processing). Example: Raw
> time-series vs. time-based rollup aggregates and compacted time-series. We
> collect raw time-series and store them into CF with FIFO compaction policy,
> periodically we run task which creates rollup aggregates and compacts
> time-series, the original raw data can be discarded after that.
> # Use it for data which can be kept entirely in a a block cache (RAM/SSD).
> Say we have local SSD (1TB) which we can use as a block cache. No need for
> compaction of a raw data at all.
> Because we do not do any real compaction, we do not use CPU and IO (disk and
> network), we do not evict hot data from a block cache. The result: improved
> throughput and latency both write and read.
> See: https://github.com/facebook/rocksdb/wiki/FIFO-compaction-style
> h3. To enable FIFO compaction policy
> For table:
> {code}
> HTableDescriptor desc = new HTableDescriptor(tableName);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> For CF:
> {code}
> HColumnDescriptor desc = new HColumnDescriptor(family);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> Although region splitting is supported, for optimal performance it should be
> disabled, either by setting explicitly DisabledRegionSplitPolicy or by
> setting ConstantSizeRegionSplitPolicy and very large max region size. You
> will have to increase to a very large number store's blocking file number :
> *hbase.hstore.blockingStoreFiles* as well.
>
> h3. Limitations
> Do not use FIFO compaction if :
> * Table/CF has MIN_VERSION > 0
> * Table/CF has TTL = FOREVER (HColumnDescriptor.DEFAULT_TTL)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14468) Compaction improvements: FIFO compaction policy
[
https://issues.apache.org/jira/browse/HBASE-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15086210#comment-15086210
]
Lars Hofhansl commented on HBASE-14468:
---
Hmm... We have the CompactionChecker in the RS, which wakes up every 10s. But I
notice it actually does work only every 1000's iteration by default.
Sorry for the false alarm.
The code does look a bit funky (HRegionServer.CompactionChecker.chore()):
If I read this right, then all stores that need compaction will trigger a
compaction on the very same iteration (assuming all have the same default
compaction multiplier). How that not lead to compaction storms? But that is
unrelated to this.
> Compaction improvements: FIFO compaction policy
> ---
>
> Key: HBASE-14468
> URL: https://issues.apache.org/jira/browse/HBASE-14468
> Project: HBase
> Issue Type: Improvement
> Components: Compaction, Performance
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14468-0.98-v2.txt, 14468-0.98.txt, HBASE-14468-v1.patch,
> HBASE-14468-v10.patch, HBASE-14468-v2.patch, HBASE-14468-v3.patch,
> HBASE-14468-v4.patch, HBASE-14468-v5.patch, HBASE-14468-v6.patch,
> HBASE-14468-v7.patch, HBASE-14468-v8.patch, HBASE-14468-v9.patch,
> HBASE-14468.add.patch
>
>
> h2. FIFO Compaction
> h3. Introduction
> FIFO compaction policy selects only files which have all cells expired. The
> column family MUST have non-default TTL.
> Essentially, FIFO compactor does only one job: collects expired store files.
> These are some applications which could benefit the most:
> # Use it for very high volume raw data which has low TTL and which is the
> source of another data (after additional processing). Example: Raw
> time-series vs. time-based rollup aggregates and compacted time-series. We
> collect raw time-series and store them into CF with FIFO compaction policy,
> periodically we run task which creates rollup aggregates and compacts
> time-series, the original raw data can be discarded after that.
> # Use it for data which can be kept entirely in a a block cache (RAM/SSD).
> Say we have local SSD (1TB) which we can use as a block cache. No need for
> compaction of a raw data at all.
> Because we do not do any real compaction, we do not use CPU and IO (disk and
> network), we do not evict hot data from a block cache. The result: improved
> throughput and latency both write and read.
> See: https://github.com/facebook/rocksdb/wiki/FIFO-compaction-style
> h3. To enable FIFO compaction policy
> For table:
> {code}
> HTableDescriptor desc = new HTableDescriptor(tableName);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> For CF:
> {code}
> HColumnDescriptor desc = new HColumnDescriptor(family);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> Although region splitting is supported, for optimal performance it should be
> disabled, either by setting explicitly DisabledRegionSplitPolicy or by
> setting ConstantSizeRegionSplitPolicy and very large max region size. You
> will have to increase to a very large number store's blocking file number :
> *hbase.hstore.blockingStoreFiles* as well.
>
> h3. Limitations
> Do not use FIFO compaction if :
> * Table/CF has MIN_VERSION > 0
> * Table/CF has TTL = FOREVER (HColumnDescriptor.DEFAULT_TTL)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15086409#comment-15086409 ] Lars Hofhansl commented on HBASE-14213: --- :) Thanks [~busbey]! > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15086420#comment-15086420 ] Lars Hofhansl commented on HBASE-14213: --- Also... Can I/we manually make a good faith best-effort to update the LICENSE/NOTICE file? Since this is the last release, it does not need to be automated. I think a good faith effort should be good as far as Apache goes. > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15086857#comment-15086857 ] Lars Hofhansl commented on HBASE-14213: --- 0.94 has close 900 files that still include the copyright header. I think I can automate most with mix of find, grep, and sed. The LICENSE/NOTICE file I should be able to just copy from 0.98. > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-14213: -- Attachment: 14213-part1.txt -part1: clean all Java and XML files > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > Attachments: 14213-part1.txt > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-14213: -- Attachment: 14213-part3.sh -part3 shell scripts > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > Attachments: 14213-part1.txt, 14213-part2.txt, 14213-part3.sh > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15086899#comment-15086899 ] Lars Hofhansl commented on HBASE-14213: --- Accidentally added my helper scripts to the patches, but that rest should be OK. > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > Attachments: 14213-part1.txt, 14213-part2.txt, 14213-part3.sh, > 14213-part4.sh, 14213-part5.sh > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14468) Compaction improvements: FIFO compaction policy
[
https://issues.apache.org/jira/browse/HBASE-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-14468:
--
Fix Version/s: 0.98.17
> Compaction improvements: FIFO compaction policy
> ---
>
> Key: HBASE-14468
> URL: https://issues.apache.org/jira/browse/HBASE-14468
> Project: HBase
> Issue Type: Improvement
> Components: Compaction, Performance
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.17
>
> Attachments: 14468-0.98-v2.txt, 14468-0.98.txt, HBASE-14468-v1.patch,
> HBASE-14468-v10.patch, HBASE-14468-v2.patch, HBASE-14468-v3.patch,
> HBASE-14468-v4.patch, HBASE-14468-v5.patch, HBASE-14468-v6.patch,
> HBASE-14468-v7.patch, HBASE-14468-v8.patch, HBASE-14468-v9.patch,
> HBASE-14468.add.patch
>
>
> h2. FIFO Compaction
> h3. Introduction
> FIFO compaction policy selects only files which have all cells expired. The
> column family MUST have non-default TTL.
> Essentially, FIFO compactor does only one job: collects expired store files.
> These are some applications which could benefit the most:
> # Use it for very high volume raw data which has low TTL and which is the
> source of another data (after additional processing). Example: Raw
> time-series vs. time-based rollup aggregates and compacted time-series. We
> collect raw time-series and store them into CF with FIFO compaction policy,
> periodically we run task which creates rollup aggregates and compacts
> time-series, the original raw data can be discarded after that.
> # Use it for data which can be kept entirely in a a block cache (RAM/SSD).
> Say we have local SSD (1TB) which we can use as a block cache. No need for
> compaction of a raw data at all.
> Because we do not do any real compaction, we do not use CPU and IO (disk and
> network), we do not evict hot data from a block cache. The result: improved
> throughput and latency both write and read.
> See: https://github.com/facebook/rocksdb/wiki/FIFO-compaction-style
> h3. To enable FIFO compaction policy
> For table:
> {code}
> HTableDescriptor desc = new HTableDescriptor(tableName);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> For CF:
> {code}
> HColumnDescriptor desc = new HColumnDescriptor(family);
>
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY,
> FIFOCompactionPolicy.class.getName());
> {code}
> From HBase shell:
> {code}
> create 'x',{NAME=>'y', TTL=>'30'}, {CONFIGURATION =>
> {'hbase.hstore.defaultengine.compactionpolicy.class' =>
> 'org.apache.hadoop.hbase.regionserver.compactions.FIFOCompactionPolicy',
> 'hbase.hstore.blockingStoreFiles' => 1000}}
> {code}
> Although region splitting is supported, for optimal performance it should be
> disabled, either by setting explicitly DisabledRegionSplitPolicy or by
> setting ConstantSizeRegionSplitPolicy and very large max region size. You
> will have to increase to a very large number store's blocking file number :
> *hbase.hstore.blockingStoreFiles* as well (there is a sanity check on
> table/column family configuration in case of FIFO compaction and minimum
> value for number of blocking file is 1000).
>
> h3. Limitations
> Do not use FIFO compaction if :
> * Table/CF has MIN_VERSION > 0
> * Table/CF has TTL = FOREVER (HColumnDescriptor.DEFAULT_TTL)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15086941#comment-15086941 ] Lars Hofhansl edited comment on HBASE-14213 at 1/7/16 6:56 AM: --- LICENSE and NOTICE (copied from 0.98, don't think we changed dependencies) [~busbey], [~stack], is that enough good faith to ship the last 0.94 release? was (Author: lhofhansl): LICENSE and NOTICE [~busbey], [~stack], is that enough good faith to ship the last 0.94 release? > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > Attachments: 14213-LICENSE.txt, 14213-combined.txt, 14213-part1.txt, > 14213-part2.txt, 14213-part3.sh, 14213-part4.sh, 14213-part5.sh > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15087879#comment-15087879 ] Lars Hofhansl commented on HBASE-14213: --- Awesome. Thanks [~busbey]. I owe you a coffee next time you're around! I only had to add a license to hadoop-metrics.properties (was flagged by rat) Looked at the generated tarballs, looks all good. Going to commit and spin a release! Thanks again. > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > Attachments: 14213-LICENSE.txt, 14213-combined.txt, 14213-part1.txt, > 14213-part2.txt, 14213-part3.sh, 14213-part4.sh, 14213-part5.sh, > HBASE-14213.1.0.94.patch > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[
https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088517#comment-15088517
]
Lars Hofhansl commented on HBASE-14213:
---
Yesterday I only built with "-DskipTests install". Seems when actually building
the assembly ("assembly:single") the build is currently broken. Patch
forthcoming.
> Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in
> artifacts for 0.94
> --
>
> Key: HBASE-14213
> URL: https://issues.apache.org/jira/browse/HBASE-14213
> Project: HBase
> Issue Type: Task
> Components: build
>Reporter: Nick Dimiduk
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 0.94.28
>
> Attachments: 14213-LICENSE.txt, 14213-combined.txt, 14213-part1.txt,
> 14213-part2.txt, 14213-part3.sh, 14213-part4.sh, 14213-part5.sh,
> HBASE-14213.1.0.94.patch
>
>
> From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took
> the liberty of assigning to [~busbey].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[
https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088517#comment-15088517
]
Lars Hofhansl edited comment on HBASE-14213 at 1/8/16 1:10 AM:
---
Yesterday I only built with "-DskipTests install -Prelease". Seems when
actually building the assembly ("assembly:single") the build is currently
broken. Patch forthcoming.
was (Author: lhofhansl):
Yesterday I only built with "-DskipTests install". Seems when actually building
the assembly ("assembly:single") the build is currently broken. Patch
forthcoming.
> Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in
> artifacts for 0.94
> --
>
> Key: HBASE-14213
> URL: https://issues.apache.org/jira/browse/HBASE-14213
> Project: HBase
> Issue Type: Task
> Components: build
>Reporter: Nick Dimiduk
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 0.94.28
>
> Attachments: 14213-LICENSE.txt, 14213-combined.txt, 14213-part1.txt,
> 14213-part2.txt, 14213-part3.sh, 14213-part4.sh, 14213-part5.sh,
> HBASE-14213.1.0.94.patch
>
>
> From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took
> the liberty of assigning to [~busbey].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088545#comment-15088545 ] Lars Hofhansl commented on HBASE-14213: --- Hmm... When fixed I get a 250MB tarball (was 57MB) before. Somehow devapidoc and testapidoc is included, but wasn't before. > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > Attachments: 14213-LICENSE.txt, 14213-combined.txt, 14213-part1.txt, > 14213-part2.txt, 14213-part3.sh, 14213-part4.sh, 14213-part5.sh, > HBASE-14213.1.0.94.patch > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-14213: -- Attachment: 14213-addendum.txt This fixes the build issues for me: # not using shade plugin # not referring to non-existent supplemental-model.xml # not trying to filter missing LEGAL file I still see the large tarball, but I saw this even when I revert this patch. So more work to do. > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > Attachments: 14213-LICENSE.txt, 14213-addendum.txt, > 14213-combined.txt, 14213-part1.txt, 14213-part2.txt, 14213-part3.sh, > 14213-part4.sh, 14213-part5.sh, HBASE-14213.1.0.94.patch > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088800#comment-15088800 ] Lars Hofhansl commented on HBASE-14213: --- Going to commit the addendum, unless I hear objections. > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > Attachments: 14213-LICENSE.txt, 14213-addendum.txt, > 14213-combined.txt, 14213-part1.txt, 14213-part2.txt, 14213-part3.sh, > 14213-part4.sh, 14213-part5.sh, HBASE-14213.1.0.94.patch > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14747) Make it possible to build Javadoc and xref reports for 0.94 again
[ https://issues.apache.org/jira/browse/HBASE-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088813#comment-15088813 ] Lars Hofhansl commented on HBASE-14747: --- This actually increases the size of the 0.94 tarball from 57MB to 255MB (must larger than any other - earlier or later - version). I'm going to revert unless I hear a compelling reason not to. [~misty], [~stack]. > Make it possible to build Javadoc and xref reports for 0.94 again > - > > Key: HBASE-14747 > URL: https://issues.apache.org/jira/browse/HBASE-14747 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.27 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 0.94.28 > > Attachments: HBASE-14747-0.94.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088815#comment-15088815 ] Lars Hofhansl commented on HBASE-14213: --- Turns out the large tarball size is due to HBASE-14747. > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > Attachments: 14213-LICENSE.txt, 14213-addendum.txt, > 14213-combined.txt, 14213-part1.txt, 14213-part2.txt, 14213-part3.sh, > 14213-part4.sh, 14213-part5.sh, HBASE-14213.1.0.94.patch > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089656#comment-15089656 ] Lars Hofhansl commented on HBASE-14213: --- [~busbey], will this still do what you intended with the addendum patch? > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > Attachments: 14213-LICENSE.txt, 14213-addendum.txt, > 14213-combined.txt, 14213-part1.txt, 14213-part2.txt, 14213-part3.sh, > 14213-part4.sh, 14213-part5.sh, HBASE-14213.1.0.94.patch > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14747) Make it possible to build Javadoc and xref reports for 0.94 again
[ https://issues.apache.org/jira/browse/HBASE-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089658#comment-15089658 ] Lars Hofhansl edited comment on HBASE-14747 at 1/8/16 6:35 PM: --- Any objections to reverting for now? Is there something else I can do? Will revert if I do not hear anything today :) was (Author: lhofhansl): Any objections to reverting for now. Is there something else I can do. Will revert if I do not hear anything today :) > Make it possible to build Javadoc and xref reports for 0.94 again > - > > Key: HBASE-14747 > URL: https://issues.apache.org/jira/browse/HBASE-14747 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.27 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 0.94.28 > > Attachments: HBASE-14747-0.94.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14747) Make it possible to build Javadoc and xref reports for 0.94 again
[ https://issues.apache.org/jira/browse/HBASE-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089658#comment-15089658 ] Lars Hofhansl commented on HBASE-14747: --- Any objections to reverting for now. Is there something else I can do. Will revert if I do not hear anything today :) > Make it possible to build Javadoc and xref reports for 0.94 again > - > > Key: HBASE-14747 > URL: https://issues.apache.org/jira/browse/HBASE-14747 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.27 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 0.94.28 > > Attachments: HBASE-14747-0.94.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089706#comment-15089706 ] Lars Hofhansl commented on HBASE-14213: --- Is supplemental-model the same as in 0.98? The LEGAL does not exist for some reason. I'll look again. > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > Attachments: 14213-LICENSE.txt, 14213-addendum.txt, > 14213-combined.txt, 14213-part1.txt, 14213-part2.txt, 14213-part3.sh, > 14213-part4.sh, 14213-part5.sh, HBASE-14213.1.0.94.patch > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089706#comment-15089706 ] Lars Hofhansl edited comment on HBASE-14213 at 1/8/16 6:59 PM: --- Is supplemental-model the same as in 0.98? Seems it's just missing from the 0.94 patch. The LEGAL does not exist for some reason. I'll look again. was (Author: lhofhansl): Is supplemental-model the same as in 0.98? The LEGAL does not exist for some reason. I'll look again. > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > Attachments: 14213-LICENSE.txt, 14213-addendum.txt, > 14213-combined.txt, 14213-part1.txt, 14213-part2.txt, 14213-part3.sh, > 14213-part4.sh, 14213-part5.sh, HBASE-14213.1.0.94.patch > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089706#comment-15089706 ] Lars Hofhansl edited comment on HBASE-14213 at 1/8/16 6:59 PM: --- Is supplemental-model the same as in 0.98? Seems it's just missing from the 0.94 patch. The jruby LEGAL file does not exist for some reason. I'll look again. was (Author: lhofhansl): Is supplemental-model the same as in 0.98? Seems it's just missing from the 0.94 patch. The LEGAL does not exist for some reason. I'll look again. > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > Attachments: 14213-LICENSE.txt, 14213-addendum.txt, > 14213-combined.txt, 14213-part1.txt, 14213-part2.txt, 14213-part3.sh, > 14213-part4.sh, 14213-part5.sh, HBASE-14213.1.0.94.patch > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14747) Make it possible to build Javadoc and xref reports for 0.94 again
[ https://issues.apache.org/jira/browse/HBASE-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089721#comment-15089721 ] Lars Hofhansl commented on HBASE-14747: --- It'll make it like the previous tarballs, which had everything. OK, I'll revert. We can always reapply. > Make it possible to build Javadoc and xref reports for 0.94 again > - > > Key: HBASE-14747 > URL: https://issues.apache.org/jira/browse/HBASE-14747 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.27 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 0.94.28 > > Attachments: HBASE-14747-0.94.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089762#comment-15089762 ] Lars Hofhansl commented on HBASE-14213: --- Hmm... You're right. Not sure what happened. I'll revert what I applied and try again. > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > Attachments: 14213-LICENSE.txt, 14213-addendum.txt, > 14213-combined.txt, 14213-part1.txt, 14213-part2.txt, 14213-part3.sh, > 14213-part4.sh, 14213-part5.sh, HBASE-14213.1.0.94.patch > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089790#comment-15089790 ] Lars Hofhansl commented on HBASE-14213: --- Reverted locally and reapplied the patch and did indeed see these files added. Too weird. Sorry about this. Addendum in a minute just running a build now. > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > Attachments: 14213-LICENSE.txt, 14213-addendum.txt, > 14213-combined.txt, 14213-part1.txt, 14213-part2.txt, 14213-part3.sh, > 14213-part4.sh, 14213-part5.sh, HBASE-14213.1.0.94.patch > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089813#comment-15089813 ] Lars Hofhansl commented on HBASE-14213: --- rat check fails on conf/hadoop-metrics.properties, I added the license header back in, unless that's unintended. > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > Attachments: 14213-LICENSE.txt, 14213-addendum.txt, > 14213-combined.txt, 14213-part1.txt, 14213-part2.txt, 14213-part3.sh, > 14213-part4.sh, 14213-part5.sh, HBASE-14213.1.0.94.patch > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089829#comment-15089829 ] Lars Hofhansl commented on HBASE-14213: --- Added the missing files. Added license header back to conf/hadoop-metrics.properties. All compiles now. > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > Attachments: 14213-LICENSE.txt, 14213-addendum.txt, > 14213-combined.txt, 14213-part1.txt, 14213-part2.txt, 14213-part3.sh, > 14213-part4.sh, 14213-part5.sh, HBASE-14213.1.0.94.patch > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089834#comment-15089834 ] Lars Hofhansl commented on HBASE-14213: --- I owe you two coffees now :) > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > Attachments: 14213-LICENSE.txt, 14213-addendum.txt, > 14213-combined.txt, 14213-part1.txt, 14213-part2.txt, 14213-part3.sh, > 14213-part4.sh, 14213-part5.sh, HBASE-14213.1.0.94.patch > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089829#comment-15089829 ] Lars Hofhansl edited comment on HBASE-14213 at 1/8/16 7:51 PM: --- Added the missing files - still not sure how I missed them. Added license header back to conf/hadoop-metrics.properties. All compiles a builds the assembly now. was (Author: lhofhansl): Added the missing files. Added license header back to conf/hadoop-metrics.properties. All compiles now. > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > Attachments: 14213-LICENSE.txt, 14213-addendum.txt, > 14213-combined.txt, 14213-part1.txt, 14213-part2.txt, 14213-part3.sh, > 14213-part4.sh, 14213-part5.sh, HBASE-14213.1.0.94.patch > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089974#comment-15089974 ] Lars Hofhansl commented on HBASE-14213: --- Arrgghh... I did run a whole build. Now the build complains about "repository.codehaus.org" not being accessible. I'll look into it. > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > Attachments: 14213-LICENSE.txt, 14213-addendum.txt, > 14213-combined.txt, 14213-part1.txt, 14213-part2.txt, 14213-part3.sh, > 14213-part4.sh, 14213-part5.sh, HBASE-14213.1.0.94.patch > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089982#comment-15089982 ] Lars Hofhansl commented on HBASE-14213: --- Everything in repository.codehaus.org has been moved to maven central, I'll remove that from the pom. Independent of this patch. > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > Attachments: 14213-LICENSE.txt, 14213-addendum.txt, > 14213-combined.txt, 14213-part1.txt, 14213-part2.txt, 14213-part3.sh, > 14213-part4.sh, 14213-part5.sh, HBASE-14213.1.0.94.patch > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15084) Remove references to repository.codehaus.org
Lars Hofhansl created HBASE-15084: - Summary: Remove references to repository.codehaus.org Key: HBASE-15084 URL: https://issues.apache.org/jira/browse/HBASE-15084 Project: HBase Issue Type: Bug Affects Versions: 0.94.27 Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.28 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089987#comment-15089987 ] Lars Hofhansl commented on HBASE-14213: --- At home I must have had stuff cached. > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > Attachments: 14213-LICENSE.txt, 14213-addendum.txt, > 14213-combined.txt, 14213-part1.txt, 14213-part2.txt, 14213-part3.sh, > 14213-part4.sh, 14213-part5.sh, HBASE-14213.1.0.94.patch > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15084) Remove references to repository.codehaus.org
[ https://issues.apache.org/jira/browse/HBASE-15084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-15084: -- Description: repository.codehause.org is not longer active. A dns-lookup reveals an alias to stop-looking-at.repository-codehaus-org :) All repositories have been moved to Maven central, so it can just removed from the pom. > Remove references to repository.codehaus.org > > > Key: HBASE-15084 > URL: https://issues.apache.org/jira/browse/HBASE-15084 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.27 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.94.28 > > > repository.codehause.org is not longer active. > A dns-lookup reveals an alias to stop-looking-at.repository-codehaus-org :) > All repositories have been moved to Maven central, so it can just removed > from the pom. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-15084) Remove references to repository.codehaus.org
[ https://issues.apache.org/jira/browse/HBASE-15084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-15084. --- Resolution: Fixed Committed to 0.94. > Remove references to repository.codehaus.org > > > Key: HBASE-15084 > URL: https://issues.apache.org/jira/browse/HBASE-15084 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.27 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.94.28 > > > repository.codehause.org is not longer active. > A dns-lookup reveals an alias to stop-looking-at.repository-codehaus-org :) > All repositories have been moved to Maven central, so it can just removed > from the pom. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14747) Make it possible to build Javadoc and xref reports for 0.94 again
[ https://issues.apache.org/jira/browse/HBASE-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15090011#comment-15090011 ] Lars Hofhansl commented on HBASE-14747: --- Turns out that all the new stuff is under docs/apidocs/src-html. Uncompressed that one directory in 1.3GB, everything else (included libraries, executables, etc) is less than 400MB. > Make it possible to build Javadoc and xref reports for 0.94 again > - > > Key: HBASE-14747 > URL: https://issues.apache.org/jira/browse/HBASE-14747 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.27 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 0.94.28 > > Attachments: HBASE-14747-0.94.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14747) Make it possible to build Javadoc and xref reports for 0.94 again
[ https://issues.apache.org/jira/browse/HBASE-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15090092#comment-15090092 ] Lars Hofhansl commented on HBASE-14747: --- Aha... We had true in there a bunch of time. That actually generate an HTML rendition of the source of _every single class_. Removing those the tarball size goes to 65MB. Unpacked with this patch I get a 300MB without 200MB. I see extra (with the patch): * testdevapidocs * testapidocs * devapidocs These three directories add up to 100MB, I don't see why we would need to ship those. > Make it possible to build Javadoc and xref reports for 0.94 again > - > > Key: HBASE-14747 > URL: https://issues.apache.org/jira/browse/HBASE-14747 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.27 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 0.94.28 > > Attachments: HBASE-14747-0.94.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14747) Make it possible to build Javadoc and xref reports for 0.94 again
[ https://issues.apache.org/jira/browse/HBASE-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-14747: -- Attachment: 14747-addendum.txt Addendum avoiding HTML rendition of every source file. > Make it possible to build Javadoc and xref reports for 0.94 again > - > > Key: HBASE-14747 > URL: https://issues.apache.org/jira/browse/HBASE-14747 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.27 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 0.94.28 > > Attachments: 14747-addendum.txt, HBASE-14747-0.94.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[
https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15090437#comment-15090437
]
Lars Hofhansl commented on HBASE-14213:
---
And home (JDK8, Maven 3.3.9) I still cannot build with this patch applied.
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-site-plugin:3.3:site (default-site) on project
hbase: failed to get report for org.apache.maven.plugins:maven-javadoc-plugin:
Failed to execute goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process
(build-legal-for-assembly) on project hbase: Unable to parse configuration of
mojo org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process for
parameter resource: Cannot find default setter in class
org.apache.maven.model.Resource -> [Help 1]
{code}
Work (JDK7, Maven 3.0.4) is fine and without patch I could build fine in both
setups.
I can't win with this one it seems :(
> Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in
> artifacts for 0.94
> --
>
> Key: HBASE-14213
> URL: https://issues.apache.org/jira/browse/HBASE-14213
> Project: HBase
> Issue Type: Task
> Components: build
>Reporter: Nick Dimiduk
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 0.94.28
>
> Attachments: 14213-LICENSE.txt, 14213-addendum.txt,
> 14213-combined.txt, 14213-part1.txt, 14213-part2.txt, 14213-part3.sh,
> 14213-part4.sh, 14213-part5.sh, HBASE-14213.1.0.94.patch
>
>
> From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took
> the liberty of assigning to [~busbey].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14213) Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in artifacts for 0.94
[ https://issues.apache.org/jira/browse/HBASE-14213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-14213: -- Attachment: 14213-addendum2.txt -addendum2 allows me to build in both environments. I do not know anything about maven, this is just poking around. > Ensure ASF policy compliant headers and correct LICENSE and NOTICE files in > artifacts for 0.94 > -- > > Key: HBASE-14213 > URL: https://issues.apache.org/jira/browse/HBASE-14213 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Sean Busbey >Priority: Blocker > Fix For: 0.94.28 > > Attachments: 14213-LICENSE.txt, 14213-addendum.txt, > 14213-addendum2.txt, 14213-combined.txt, 14213-part1.txt, 14213-part2.txt, > 14213-part3.sh, 14213-part4.sh, 14213-part5.sh, HBASE-14213.1.0.94.patch > > > From tail of thread on HBASE-14085, opening a backport ticket for 0.94. Took > the liberty of assigning to [~busbey]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
