[ https://issues.apache.org/jira/browse/HBASE-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906117#comment-15906117 ]
Tianying Chang commented on HBASE-17138: ---------------------------------------- thanks [~anoop.hbase] what do you mean by trunk patches backport? Is this Alibaba's patch list for backporting from trunk/2.0 to 1.1? So the patch list above theoretically can be applied one by one on 1.2.1 open source version? > Backport read-path offheap (HBASE-11425) to branch-1 > ---------------------------------------------------- > > Key: HBASE-17138 > URL: https://issues.apache.org/jira/browse/HBASE-17138 > Project: HBase > Issue Type: Improvement > Reporter: Yu Li > Assignee: Yu Sun > Attachments: > 0001-fix-EHB-511-Resolve-client-compatibility-issue-introduced-by-offheap-change.patch, > 0001-to-EHB-446-offheap-hfile-format-should-keep-compatible-v3.patch, > 0001-to-EHB-456-Cell-should-be-compatible-with-branch-1.1.2.patch > > > From the > [thread|http://mail-archives.apache.org/mod_mbox/hbase-user/201611.mbox/%3CCAM7-19%2Bn7cEiY4H9iLQ3N9V0NXppOPduZwk-hhgNLEaJfiV3kA%40mail.gmail.com%3E] > of sharing our experience and performance data of read-path offheap usage in > Alibaba search, we could see people are positive to have HBASE-11425 in > branch-1, so I'd like to create a JIRA and move the discussion and decision > making here. > Echoing some comments from the mail thread: > Bryan: > Is the backported patch available anywhere? If it ends up not getting > officially backported to branch-1 due to 2.0 around the corner, some of us > who build our own deploy may want to integrate into our builds > Andrew: > Yes, please, the patches will be useful to the community even if we decide > not to backport into an official 1.x release. > Enis: > I don't see any reason why we cannot backport to branch-1. > Ted: > Opening a JIRA would be fine. This makes it easier for people to obtain the > patch(es) > Nick: > From the DISCUSS thread re: EOL of 1.1, it seems we'll continue to > support 1.x releases for some time... I would guess these will be > maintained until 2.2 at least. Therefore, offheap patches that have seen > production exposure seem like a reasonable candidate for backport, perhaps in > a 1.4 or 1.5 release timeframe. > Anoop: > Because of some compatibility issues, we decide that this will be done in 2.0 > only.. Ya as Andy said, it would be great to share the 1.x backported > patches. > The following is all the jira ids we have back ported: > HBASE-10930 Change Filters and GetClosestRowBeforeTracker to work with Cells > (Ram) > HBASE-13373 Squash HFileReaderV3 together with HFileReaderV2 and > AbstractHFileReader; ditto for Scanners and BlockReader, etc. > HBASE-13429 Remove deprecated seek/reseek methods from HFileScanner. > HBASE-13450 - Purge RawBytescomparator from the writers and readers for > HBASE-10800 (Ram) > HBASE-13501 - Deprecate/Remove getComparator() in HRegionInfo. > HBASE-12048 Remove deprecated APIs from Filter. > HBASE-10800 - Use CellComparator instead of KVComparator (Ram) > HBASE-13679 Change ColumnTracker and SQM to deal with Cell instead of byte[], > int, int. > HBASE-13642 Deprecate RegionObserver#postScannerFilterRow CP hook with > byte[],int,int args in favor of taking Cell arg. > HBASE-13641 Deperecate Filter#filterRowKey(byte[] buffer, int offset, int > length) in favor of filterRowKey(Cell firstRowCell). > HBASE-13827 Delayed scanner close in KeyValueHeap and StoreScanner. > HBASE-13871 Change RegionScannerImpl to deal with Cell instead of byte[], > int, int. > HBASE-11911 Break up tests into more fine grained categories (Alex Newman) > HBASE-12059 Create hbase-annotations module > HBASE-12106 Move test annotations to test artifact (Enis Soztutar) > HBASE-13916 Create MultiByteBuffer an aggregation of ByteBuffers. > HBASE-15679 Assertion on wrong variable in > TestReplicationThrottler#testThrottling > HBASE-13931 Move Unsafe based operations to UnsafeAccess. > HBASE-12345 Unsafe based ByteBuffer Comparator. > HBASE-13998 Remove CellComparator#compareRows(byte[] left, int loffset, int > llength, byte[] right, int roffset, int rlength). > HBASE-13998 Remove CellComparator#compareRows()- Addendum to fix javadoc warn > HBASE-13579 Avoid isCellTTLExpired() for NO-TAG cases (partially backport > this patch) > HBASE-13448 New Cell implementation with cached component offsets/lengths. > HBASE-13387 Add ByteBufferedCell an extension to Cell. > HBASE-13387 Add ByteBufferedCell an extension to Cell - addendum. > HBASE-12650 Move ServerName to hbase-common module (partially backport this > patch) > HBASE-12296 Filters should work with ByteBufferedCell. > HBASE-14120 ByteBufferUtils#compareTo small optimization. > HBASE-13510 - Purge ByteBloomFilter (Ram) > HBASE-13451 - Make the HFileBlockIndex blockKeys to Cells so that it could be > easy to use in the CellComparators (Ram) > HBASE-13614 - Avoid temp KeyOnlyKeyValue temp objects creations in read hot > path (Ram) > HBASE-13939 - Make HFileReaderImpl.getFirstKeyInBlock() to return a Cell (Ram) > HBASE-13307 Making methods under ScannerV2#next inlineable, faster > HBASE-14020 Unsafe based optimized write in ByteBufferOutputStream. > HBASE-13977 - Convert getKey and related APIs to Cell (Ram) > HBASE-11927 Use Native Hadoop Library for HFile checksum. (Apekshit) > HBASE-12213 HFileBlock backed by Array of ByteBuffers (Ram) > HBASE-12084 Remove deprecated APIs from Result. > HBASE-12084 Remove deprecated APIs from Result - shell addendum > HBASE-13754 Allow non KeyValue Cell types also to oswrite. > HBASE-14047 - Cleanup deprecated APIs from Cell class (Ashish Singhi) > HBASE-13817 ByteBufferOuputStream - add writeInt support. > HBASE-12374 Change DBEs to work with new BB based cell. > HBASE-14116 Change ByteBuff.getXXXStrictlyForward to relative position based > reads > HBASE-14073 TestRemoteTable.testDelete failed in the latest trunk > code.(Jingcheng) > HBASE-13926 Close the scanner only after Call#setResponse. > HBASE-12295 Prevent block eviction under us if reads are in progress from the > BBs (Ram) > HBASE-12295 - Addendum for multiGets to add the call back(Ram) > HBASE-14001 Optimize write(OutputStream out, boolean withTags) for > SizeCachedNoTagsKeyValue. > HBASE-14063 Use BufferBackedCell in read path after HBASE-12213 and > HBASE-12295 (ram) > HBASE-14202 Reduce garbage we create. > HBASE-14099 StoreFile.passesKeyRangeFilter need not create Cells from the > Scan's start and stop Row (Ram) > HBASE-14144 - Bloomfilter path to work with Byte buffered cells (Ram) > HBASE-14395 Short circuit last byte check in CellUtil#matchingXXX methods for > ByteBufferedCells. > HBASE-12298 Support BB usage in PrefixTree (Ram) > HBASE-14398 - Create the fake keys required in the scan path to avoid copy to > byte[] (Ram) > HBASE-14480 Small optimization in SingleByteBuff. > HBASE-14590 Shorten ByteBufferedCell#getXXXPositionInByteBuffer method name > (Anoop Sam John) > HBASE-14636 Clear HFileScannerImpl#prevBlocks in between Compaction flow. > HBASE-14832 Ensure write paths work with ByteBufferedCells in case of > compaction (Ram) > HBASE-14524 Short-circuit comparison of rows in CellComparator. (Lars Francke) > HBASE-14188 - Read path optimizations after HBASE-11425 profiling (Ram) > HBASE-14188- Read path optimizations after HBASE-11425 profiling-Addendum(Ram) > HBASE-12593 Tags to work with ByteBuffer. > HBASE-15077 Support OffheapKV write in compaction with out copying data on > heap. > HBASE-14660 AssertionError found when using offheap BucketCache with > assertion enabled (ram) > HBASE-15076 Add getScanner(Scan scan, List<KeyValueScanner> > additionalScanners) API into Region interface (Anoop Sam John > HBASE-15735(also particial HBASE-13893) Tightening of the CP contract. > HBASE-15785 Unnecessary lock in ByteBufferArray. > HBASE-15760 > TestBlockEvictionFromClient#testParallelGetsAndScanWithWrappedRegionScanner > fails in master branch (Ram) > HBASE-15379 Fake cells created in read path not implementing > SettableSequenceId > HBASE-16609 Fake cells EmptyByteBufferedCell created in read path not > implementing SettableSequenceId > HBASE-14940 Make our unsafe based ops more safe > HBASE-15063 Bug in MultiByteBuf#toBytes. (deepankar) > HBASE-15064 BufferUnderflowException after last Cell fetched from an HFile > Block served from L2 offheap cache. > HBASE-15064 BufferUnderflowException after last Cell fetched from an HFile > Block served from L2 offheap cache - Addendum. > HBASE-15253 Small bug in CellUtil.matchingRow(Cell, byte[]) (Ram) > HBASE-16704 Scan will be broken while working with DBE and > KeyValueCodecWithTags > the above jira ids are listed in chronological order I have backport,and > there are also some patch i dont list, such as: > 1. keep Cell api compatible with our existing code. > 2. hfile format related compatibility issues. > 3. client compatibility issue -- This message was sent by Atlassian JIRA (v6.3.15#6346)