[
https://issues.apache.org/jira/browse/HBASE-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089053#comment-15089053
]
Anoop Sam John commented on HBASE-15077:
----------------------------------------
[[email protected]]
ByteBufferWriter seems a better name and shorter. :-)
[~ram_krish]
Yes we have to copy it to a stream. That we can not avoid. But that copy is
direct now which copies the bytes from BB as whole in one shot into the byte
array backing the stream. Previously, it was happening like byte by byte copy
from off heap area into a ref and then to the stream array.
Yes we dont need BufferGrabbingByteArrayOutputStream now. That was a fancy
stuff to grab the backing byte[] of the stream.Now our own BAOS gives it with
an API. Its our own usage.
We have options to enable compression and/or encryption of the data in HFile.
For that to work we might need this on heap streams I believe. So not sure
whether we can do direct streaming. Any way not scope of this Jira. Let us
investigate it later.
> Support OffheapKV write in compaction with out copying data on heap
> -------------------------------------------------------------------
>
> Key: HBASE-15077
> URL: https://issues.apache.org/jira/browse/HBASE-15077
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver, Scanners
> Reporter: Anoop Sam John
> Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15077.patch
>
>
> HBASE-14832 is not enough to handle this. Doing the remaining needed here.
> {code}
> if (cell instanceof ByteBufferedCell) {
> 890 out.writeShort(rowLen);
> 891 ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell)
> cell).getRowByteBuffer(),
> 892 ((ByteBufferedCell) cell).getRowPosition(), rowLen);
> 893 out.writeByte(fLen);
> 894 ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell)
> cell).getFamilyByteBuffer(),
> 895 ((ByteBufferedCell) cell).getFamilyPosition(), fLen);
> 896 ByteBufferUtils.copyBufferToStream(out, ((ByteBufferedCell)
> cell).getQualifierByteBuffer(),
> 897 ((ByteBufferedCell) cell).getQualifierPosition(), qLen);
> {code}
> We have done this but it is not really helping us!
> In ByteBufferUtils#copyBufferToStream
> {code}
> public static void copyBufferToStream(OutputStream out, ByteBuffer in,
> int offset, int length) throws IOException {
> if (in.hasArray()) {
> out.write(in.array(), in.arrayOffset() + offset,
> length);
> } else {
> for (int i = 0; i < length; ++i) {
> out.write(toByte(in, offset + i));
> }
> }
> }
> {code}
> So for DBB it is so costly op writing byte by byte reading each to on heap.
> Even if we use writeByteBuffer(OutputStream out, ByteBuffer b, int offset,
> int length), it won't help us as the underlying stream is a
> ByteArrayOutputStream and so we will end up in copying.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
