[
https://issues.apache.org/jira/browse/HBASE-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906829#comment-15906829
]
Yu Li commented on HBASE-17138:
-------------------------------
bq. So if ur cluster is having custom CP/Filter, u might have to change the
impl code (new API implementation) and recompile and use that new jar. I
believe Alibaba also had similar case.
True story, some more work like HBASE-16626 and cp changes, case by case though.
bq. 1.2.1 will be more closer to 1.1 than 2.0
Probably, but it's almost impossible for us to fully upload our code base to
github (legal stuff you know...), and if we share the patches one by one,
there'll be 70+ to generate and w/o git (cherry-pick/rebase) I'm afraid they're
not easy to apply.
Sorry that we're short of hand to complete this backport to branch-1 since we
have a tight schedule to meet challenges of 2017 Singles' Day (while the work
listed here finished in Sep. last year), but still we hope listing out the
JIRAs and giving some proof that a backport can work well in production could
help others who want this feature ahead of time (will be officially released in
2.0).
OTOH, I'd strongly suggest to spend enough time on the backport -- read and
understand the purpose of each JIRA listed -- to make sure of the quality and
capability of handling problems if any.
> Backport read-path offheap (HBASE-11425) to branch-1
> ----------------------------------------------------
>
> Key: HBASE-17138
> URL: https://issues.apache.org/jira/browse/HBASE-17138
> Project: HBase
> Issue Type: Improvement
> Reporter: Yu Li
> Assignee: Yu Sun
> Attachments:
> 0001-fix-EHB-511-Resolve-client-compatibility-issue-introduced-by-offheap-change.patch,
> 0001-to-EHB-446-offheap-hfile-format-should-keep-compatible-v3.patch,
> 0001-to-EHB-456-Cell-should-be-compatible-with-branch-1.1.2.patch
>
>
> From the
> [thread|http://mail-archives.apache.org/mod_mbox/hbase-user/201611.mbox/%3CCAM7-19%2Bn7cEiY4H9iLQ3N9V0NXppOPduZwk-hhgNLEaJfiV3kA%40mail.gmail.com%3E]
> of sharing our experience and performance data of read-path offheap usage in
> Alibaba search, we could see people are positive to have HBASE-11425 in
> branch-1, so I'd like to create a JIRA and move the discussion and decision
> making here.
> Echoing some comments from the mail thread:
> Bryan:
> Is the backported patch available anywhere? If it ends up not getting
> officially backported to branch-1 due to 2.0 around the corner, some of us
> who build our own deploy may want to integrate into our builds
> Andrew:
> Yes, please, the patches will be useful to the community even if we decide
> not to backport into an official 1.x release.
> Enis:
> I don't see any reason why we cannot backport to branch-1.
> Ted:
> Opening a JIRA would be fine. This makes it easier for people to obtain the
> patch(es)
> Nick:
> From the DISCUSS thread re: EOL of 1.1, it seems we'll continue to
> support 1.x releases for some time... I would guess these will be
> maintained until 2.2 at least. Therefore, offheap patches that have seen
> production exposure seem like a reasonable candidate for backport, perhaps in
> a 1.4 or 1.5 release timeframe.
> Anoop:
> Because of some compatibility issues, we decide that this will be done in 2.0
> only.. Ya as Andy said, it would be great to share the 1.x backported
> patches.
> The following is all the jira ids we have back ported:
> HBASE-10930 Change Filters and GetClosestRowBeforeTracker to work with Cells
> (Ram)
> HBASE-13373 Squash HFileReaderV3 together with HFileReaderV2 and
> AbstractHFileReader; ditto for Scanners and BlockReader, etc.
> HBASE-13429 Remove deprecated seek/reseek methods from HFileScanner.
> HBASE-13450 - Purge RawBytescomparator from the writers and readers for
> HBASE-10800 (Ram)
> HBASE-13501 - Deprecate/Remove getComparator() in HRegionInfo.
> HBASE-12048 Remove deprecated APIs from Filter.
> HBASE-10800 - Use CellComparator instead of KVComparator (Ram)
> HBASE-13679 Change ColumnTracker and SQM to deal with Cell instead of byte[],
> int, int.
> HBASE-13642 Deprecate RegionObserver#postScannerFilterRow CP hook with
> byte[],int,int args in favor of taking Cell arg.
> HBASE-13641 Deperecate Filter#filterRowKey(byte[] buffer, int offset, int
> length) in favor of filterRowKey(Cell firstRowCell).
> HBASE-13827 Delayed scanner close in KeyValueHeap and StoreScanner.
> HBASE-13871 Change RegionScannerImpl to deal with Cell instead of byte[],
> int, int.
> HBASE-11911 Break up tests into more fine grained categories (Alex Newman)
> HBASE-12059 Create hbase-annotations module
> HBASE-12106 Move test annotations to test artifact (Enis Soztutar)
> HBASE-13916 Create MultiByteBuffer an aggregation of ByteBuffers.
> HBASE-15679 Assertion on wrong variable in
> TestReplicationThrottler#testThrottling
> HBASE-13931 Move Unsafe based operations to UnsafeAccess.
> HBASE-12345 Unsafe based ByteBuffer Comparator.
> HBASE-13998 Remove CellComparator#compareRows(byte[] left, int loffset, int
> llength, byte[] right, int roffset, int rlength).
> HBASE-13998 Remove CellComparator#compareRows()- Addendum to fix javadoc warn
> HBASE-13579 Avoid isCellTTLExpired() for NO-TAG cases (partially backport
> this patch)
> HBASE-13448 New Cell implementation with cached component offsets/lengths.
> HBASE-13387 Add ByteBufferedCell an extension to Cell.
> HBASE-13387 Add ByteBufferedCell an extension to Cell - addendum.
> HBASE-12650 Move ServerName to hbase-common module (partially backport this
> patch)
> HBASE-12296 Filters should work with ByteBufferedCell.
> HBASE-14120 ByteBufferUtils#compareTo small optimization.
> HBASE-13510 - Purge ByteBloomFilter (Ram)
> HBASE-13451 - Make the HFileBlockIndex blockKeys to Cells so that it could be
> easy to use in the CellComparators (Ram)
> HBASE-13614 - Avoid temp KeyOnlyKeyValue temp objects creations in read hot
> path (Ram)
> HBASE-13939 - Make HFileReaderImpl.getFirstKeyInBlock() to return a Cell (Ram)
> HBASE-13307 Making methods under ScannerV2#next inlineable, faster
> HBASE-14020 Unsafe based optimized write in ByteBufferOutputStream.
> HBASE-13977 - Convert getKey and related APIs to Cell (Ram)
> HBASE-11927 Use Native Hadoop Library for HFile checksum. (Apekshit)
> HBASE-12213 HFileBlock backed by Array of ByteBuffers (Ram)
> HBASE-12084 Remove deprecated APIs from Result.
> HBASE-12084 Remove deprecated APIs from Result - shell addendum
> HBASE-13754 Allow non KeyValue Cell types also to oswrite.
> HBASE-14047 - Cleanup deprecated APIs from Cell class (Ashish Singhi)
> HBASE-13817 ByteBufferOuputStream - add writeInt support.
> HBASE-12374 Change DBEs to work with new BB based cell.
> HBASE-14116 Change ByteBuff.getXXXStrictlyForward to relative position based
> reads
> HBASE-14073 TestRemoteTable.testDelete failed in the latest trunk
> code.(Jingcheng)
> HBASE-13926 Close the scanner only after Call#setResponse.
> HBASE-12295 Prevent block eviction under us if reads are in progress from the
> BBs (Ram)
> HBASE-12295 - Addendum for multiGets to add the call back(Ram)
> HBASE-14001 Optimize write(OutputStream out, boolean withTags) for
> SizeCachedNoTagsKeyValue.
> HBASE-14063 Use BufferBackedCell in read path after HBASE-12213 and
> HBASE-12295 (ram)
> HBASE-14202 Reduce garbage we create.
> HBASE-14099 StoreFile.passesKeyRangeFilter need not create Cells from the
> Scan's start and stop Row (Ram)
> HBASE-14144 - Bloomfilter path to work with Byte buffered cells (Ram)
> HBASE-14395 Short circuit last byte check in CellUtil#matchingXXX methods for
> ByteBufferedCells.
> HBASE-12298 Support BB usage in PrefixTree (Ram)
> HBASE-14398 - Create the fake keys required in the scan path to avoid copy to
> byte[] (Ram)
> HBASE-14480 Small optimization in SingleByteBuff.
> HBASE-14590 Shorten ByteBufferedCell#getXXXPositionInByteBuffer method name
> (Anoop Sam John)
> HBASE-14636 Clear HFileScannerImpl#prevBlocks in between Compaction flow.
> HBASE-14832 Ensure write paths work with ByteBufferedCells in case of
> compaction (Ram)
> HBASE-14524 Short-circuit comparison of rows in CellComparator. (Lars Francke)
> HBASE-14188 - Read path optimizations after HBASE-11425 profiling (Ram)
> HBASE-14188- Read path optimizations after HBASE-11425 profiling-Addendum(Ram)
> HBASE-12593 Tags to work with ByteBuffer.
> HBASE-15077 Support OffheapKV write in compaction with out copying data on
> heap.
> HBASE-14660 AssertionError found when using offheap BucketCache with
> assertion enabled (ram)
> HBASE-15076 Add getScanner(Scan scan, List<KeyValueScanner>
> additionalScanners) API into Region interface (Anoop Sam John
> HBASE-15735(also particial HBASE-13893) Tightening of the CP contract.
> HBASE-15785 Unnecessary lock in ByteBufferArray.
> HBASE-15760
> TestBlockEvictionFromClient#testParallelGetsAndScanWithWrappedRegionScanner
> fails in master branch (Ram)
> HBASE-15379 Fake cells created in read path not implementing
> SettableSequenceId
> HBASE-16609 Fake cells EmptyByteBufferedCell created in read path not
> implementing SettableSequenceId
> HBASE-14940 Make our unsafe based ops more safe
> HBASE-15063 Bug in MultiByteBuf#toBytes. (deepankar)
> HBASE-15064 BufferUnderflowException after last Cell fetched from an HFile
> Block served from L2 offheap cache.
> HBASE-15064 BufferUnderflowException after last Cell fetched from an HFile
> Block served from L2 offheap cache - Addendum.
> HBASE-15253 Small bug in CellUtil.matchingRow(Cell, byte[]) (Ram)
> HBASE-16704 Scan will be broken while working with DBE and
> KeyValueCodecWithTags
> the above jira ids are listed in chronological order I have backport，and
> there are also some patch i dont list, such as:
> 1. keep Cell api compatible with our existing code.
> 2. hfile format related compatibility issues.
> 3. client compatibility issue
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
