[jira] [Commented] (HBASE-10835) DBE encode path improvements
[ https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14006371#comment-14006371 ] Hudson commented on HBASE-10835: FAILURE: Integrated in HBase-TRUNK #5137 (See [https://builds.apache.org/job/HBase-TRUNK/5137/]) HBASE-10835 DBE encode path improvements.(Anoop) (anoopsamjohn: rev 53513dcb452e104bbfd71819054bf4d68808f731) * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/EncodedDataBlock.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockEncodingContext.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/EncodingState.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/CopyKeyDataBlockEncoder.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV3.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDefaultEncodingContext.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DiffKeyDeltaEncoder.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java * hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/encode/EncoderFactory.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestDataBlockEncoders.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoder.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestSeekToBlockWithEncoders.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoderImpl.java * hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/PrefixTreeCodec.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/NoOpDataBlockEncoder.java * hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/encode/EncoderPoolImpl.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestPrefixTreeEncoding.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java > DBE encode path improvements > > > Key: HBASE-10835 > URL: https://issues.apache.org/jira/browse/HBASE-10835 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-10835.patch, HBASE-10835_V2.patch, > HBASE-10835_V3.patch, HBASE-10835_V4.patch, HBASE-10835_V5.patch > > > Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder. > Encoder again reads kvs one by one from the buffer and encodes and creates a > new buffer. > There is no need to have this model now. Previously we had option of no > encode in disk and encode only in cache. At that time the read buffer from a > HFile block was passed to this and encodes. > So encode cell by cell can be done now. Making this change will need us to > have a NoOp DBE impl which just do the write of a cell as it is with out any > encoding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10835) DBE encode path improvements
[ https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14004908#comment-14004908 ] Anoop Sam John commented on HBASE-10835: I was/am waiting for the SVN-> GIT migration Stack.. Thanks.. > DBE encode path improvements > > > Key: HBASE-10835 > URL: https://issues.apache.org/jira/browse/HBASE-10835 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-10835.patch, HBASE-10835_V2.patch, > HBASE-10835_V3.patch, HBASE-10835_V4.patch, HBASE-10835_V5.patch > > > Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder. > Encoder again reads kvs one by one from the buffer and encodes and creates a > new buffer. > There is no need to have this model now. Previously we had option of no > encode in disk and encode only in cache. At that time the read buffer from a > HFile block was passed to this and encodes. > So encode cell by cell can be done now. Making this change will need us to > have a NoOp DBE impl which just do the write of a cell as it is with out any > encoding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10835) DBE encode path improvements
[ https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14004352#comment-14004352 ] stack commented on HBASE-10835: --- bq. Trying to get a new QA run. Once QA report is fine, I will commit. Don't hold your breath Anoop. Maybe run local and if passes, go ahead commit (note TestHCM broke if JDK6 at mo). > DBE encode path improvements > > > Key: HBASE-10835 > URL: https://issues.apache.org/jira/browse/HBASE-10835 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-10835.patch, HBASE-10835_V2.patch, > HBASE-10835_V3.patch, HBASE-10835_V4.patch, HBASE-10835_V5.patch > > > Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder. > Encoder again reads kvs one by one from the buffer and encodes and creates a > new buffer. > There is no need to have this model now. Previously we had option of no > encode in disk and encode only in cache. At that time the read buffer from a > HFile block was passed to this and encodes. > So encode cell by cell can be done now. Making this change will need us to > have a NoOp DBE impl which just do the write of a cell as it is with out any > encoding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10835) DBE encode path improvements
[ https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14003606#comment-14003606 ] ramkrishna.s.vasudevan commented on HBASE-10835: +1. Looks great. Am ok for follow up issues mentioned over in RB. > DBE encode path improvements > > > Key: HBASE-10835 > URL: https://issues.apache.org/jira/browse/HBASE-10835 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-10835.patch, HBASE-10835_V2.patch, > HBASE-10835_V3.patch, HBASE-10835_V4.patch > > > Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder. > Encoder again reads kvs one by one from the buffer and encodes and creates a > new buffer. > There is no need to have this model now. Previously we had option of no > encode in disk and encode only in cache. At that time the read buffer from a > HFile block was passed to this and encodes. > So encode cell by cell can be done now. Making this change will need us to > have a NoOp DBE impl which just do the write of a cell as it is with out any > encoding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10835) DBE encode path improvements
[ https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000920#comment-14000920 ] stack commented on HBASE-10835: --- [~anoop.hbase] The others were introduced by me Anoop (Since fixed). Yeah to fixing above warning on commit. > DBE encode path improvements > > > Key: HBASE-10835 > URL: https://issues.apache.org/jira/browse/HBASE-10835 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-10835.patch, HBASE-10835_V2.patch, > HBASE-10835_V3.patch, HBASE-10835_V4.patch > > > Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder. > Encoder again reads kvs one by one from the buffer and encodes and creates a > new buffer. > There is no need to have this model now. Previously we had option of no > encode in disk and encode only in cache. At that time the read buffer from a > HFile block was passed to this and encodes. > So encode cell by cell can be done now. Making this change will need us to > have a NoOp DBE impl which just do the write of a cell as it is with out any > encoding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10835) DBE encode path improvements
[
https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000626#comment-14000626
]
Anoop Sam John commented on HBASE-10835:
{code}
5 warnings
[WARNING] Javadoc Warnings
[WARNING] javadoc: warning - Multiple sources of package comments found for
package "org.apache.hadoop.hbase.io.hfile"
[WARNING]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoder.java:47:
warning - Tag @link: can't find endBlockEncoding(HFileBlockEncodingContext,
DataOutputStream, byte[]) in
org.apache.hadoop.hbase.io.hfile.HFileDataBlockEncoder
[WARNING]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/package-info.java:119:
warning - Tag @link: reference not found:
CacheConfig#SLAB_CACHE_OFFHEAP_PERCENTAGE_KEY
[WARNING]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/package-info.java:119:
warning - Tag @link: reference not found: HConstants#HFILE_BLOCK_CACHE_SIZE_KEY
[WARNING]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/package-info.java:119:
warning - Tag @link: reference not found:
CacheConfig#BUCKET_CACHE_COMBINED_PERCENTAGE_KEY
[INFO]
{code}
Out of this only one warn regarding HFileDataBlockEncoder is introduced by this
patch. I can correct that on commit.
> DBE encode path improvements
>
>
> Key: HBASE-10835
> URL: https://issues.apache.org/jira/browse/HBASE-10835
> Project: HBase
> Issue Type: Improvement
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 0.99.0
>
> Attachments: HBASE-10835.patch, HBASE-10835_V2.patch,
> HBASE-10835_V3.patch, HBASE-10835_V4.patch
>
>
> Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder.
> Encoder again reads kvs one by one from the buffer and encodes and creates a
> new buffer.
> There is no need to have this model now. Previously we had option of no
> encode in disk and encode only in cache. At that time the read buffer from a
> HFile block was passed to this and encodes.
> So encode cell by cell can be done now. Making this change will need us to
> have a NoOp DBE impl which just do the write of a cell as it is with out any
> encoding.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10835) DBE encode path improvements
[
https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000502#comment-14000502
]
Hadoop QA commented on HBASE-10835:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12645177/HBASE-10835_V4.patch
against trunk revision .
ATTACHMENT ID: 12645177
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 18 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 5
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9524//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9524//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9524//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9524//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9524//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9524//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9524//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9524//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9524//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9524//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9524//console
This message is automatically generated.
> DBE encode path improvements
>
>
> Key: HBASE-10835
> URL: https://issues.apache.org/jira/browse/HBASE-10835
> Project: HBase
> Issue Type: Improvement
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 0.99.0
>
> Attachments: HBASE-10835.patch, HBASE-10835_V2.patch,
> HBASE-10835_V3.patch, HBASE-10835_V4.patch
>
>
> Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder.
> Encoder again reads kvs one by one from the buffer and encodes and creates a
> new buffer.
> There is no need to have this model now. Previously we had option of no
> encode in disk and encode only in cache. At that time the read buffer from a
> HFile block was passed to this and encodes.
> So encode cell by cell can be done now. Making this change will need us to
> have a NoOp DBE impl which just do the write of a cell as it is with out any
> encoding.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10835) DBE encode path improvements
[ https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999596#comment-13999596 ] Anoop Sam John commented on HBASE-10835: [~stack] , [~ram_krish] Any further comments? I have addressed the comments or marked as TODOs for follow up issues. Can you add your +1s > DBE encode path improvements > > > Key: HBASE-10835 > URL: https://issues.apache.org/jira/browse/HBASE-10835 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-10835.patch, HBASE-10835_V2.patch, > HBASE-10835_V3.patch, HBASE-10835_V4.patch > > > Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder. > Encoder again reads kvs one by one from the buffer and encodes and creates a > new buffer. > There is no need to have this model now. Previously we had option of no > encode in disk and encode only in cache. At that time the read buffer from a > HFile block was passed to this and encodes. > So encode cell by cell can be done now. Making this change will need us to > have a NoOp DBE impl which just do the write of a cell as it is with out any > encoding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10835) DBE encode path improvements
[ https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1402#comment-1402 ] stack commented on HBASE-10835: --- +1 > DBE encode path improvements > > > Key: HBASE-10835 > URL: https://issues.apache.org/jira/browse/HBASE-10835 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-10835.patch, HBASE-10835_V2.patch, > HBASE-10835_V3.patch, HBASE-10835_V4.patch > > > Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder. > Encoder again reads kvs one by one from the buffer and encodes and creates a > new buffer. > There is no need to have this model now. Previously we had option of no > encode in disk and encode only in cache. At that time the read buffer from a > HFile block was passed to this and encodes. > So encode cell by cell can be done now. Making this change will need us to > have a NoOp DBE impl which just do the write of a cell as it is with out any > encoding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10835) DBE encode path improvements
[
https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986416#comment-13986416
]
Hadoop QA commented on HBASE-10835:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12642807/HBASE-10835_V3.patch
against trunk revision .
ATTACHMENT ID: 12642807
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 18 new
or modified tests.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestMultiParallel
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9438//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9438//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9438//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9438//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9438//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9438//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9438//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9438//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9438//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9438//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9438//console
This message is automatically generated.
> DBE encode path improvements
>
>
> Key: HBASE-10835
> URL: https://issues.apache.org/jira/browse/HBASE-10835
> Project: HBase
> Issue Type: Improvement
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 0.99.0
>
> Attachments: HBASE-10835.patch, HBASE-10835_V2.patch,
> HBASE-10835_V3.patch
>
>
> Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder.
> Encoder again reads kvs one by one from the buffer and encodes and creates a
> new buffer.
> There is no need to have this model now. Previously we had option of no
> encode in disk and encode only in cache. At that time the read buffer from a
> HFile block was passed to this and encodes.
> So encode cell by cell can be done now. Making this change will need us to
> have a NoOp DBE impl which just do the write of a cell as it is with out any
> encoding.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10835) DBE encode path improvements
[ https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986365#comment-13986365 ] Anoop Sam John commented on HBASE-10835: https://reviews.apache.org/r/20936 Also comment from Matt is fixed in this version. > DBE encode path improvements > > > Key: HBASE-10835 > URL: https://issues.apache.org/jira/browse/HBASE-10835 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-10835.patch, HBASE-10835_V2.patch, > HBASE-10835_V3.patch > > > Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder. > Encoder again reads kvs one by one from the buffer and encodes and creates a > new buffer. > There is no need to have this model now. Previously we had option of no > encode in disk and encode only in cache. At that time the read buffer from a > HFile block was passed to this and encodes. > So encode cell by cell can be done now. Making this change will need us to > have a NoOp DBE impl which just do the write of a cell as it is with out any > encoding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10835) DBE encode path improvements
[ https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986336#comment-13986336 ] Anoop Sam John commented on HBASE-10835: Sure I will put it in RB. > DBE encode path improvements > > > Key: HBASE-10835 > URL: https://issues.apache.org/jira/browse/HBASE-10835 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-10835.patch, HBASE-10835_V2.patch > > > Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder. > Encoder again reads kvs one by one from the buffer and encodes and creates a > new buffer. > There is no need to have this model now. Previously we had option of no > encode in disk and encode only in cache. At that time the read buffer from a > HFile block was passed to this and encodes. > So encode cell by cell can be done now. Making this change will need us to > have a NoOp DBE impl which just do the write of a cell as it is with out any > encoding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10835) DBE encode path improvements
[ https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986334#comment-13986334 ] Matt Corgan commented on HBASE-10835: - Anoop, i don't see anything wrong but could get a better feel for it on reviewboard. Let me know if that's possible or if I missed a review request. > DBE encode path improvements > > > Key: HBASE-10835 > URL: https://issues.apache.org/jira/browse/HBASE-10835 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-10835.patch, HBASE-10835_V2.patch > > > Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder. > Encoder again reads kvs one by one from the buffer and encodes and creates a > new buffer. > There is no need to have this model now. Previously we had option of no > encode in disk and encode only in cache. At that time the read buffer from a > HFile block was passed to this and encodes. > So encode cell by cell can be done now. Making this change will need us to > have a NoOp DBE impl which just do the write of a cell as it is with out any > encoding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10835) DBE encode path improvements
[
https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986333#comment-13986333
]
Matt Corgan commented on HBASE-10835:
-
{code}
+@InterfaceAudience.Private
+public class EncoderPoolImpl implements EncoderPool {
+
+ private List unusedEncoders = new
ArrayList();
+
+ @Override
+ public PrefixTreeEncoder checkOut(OutputStream outputStream, boolean
includeMvccVersion) {
+PrefixTreeEncoder encoder = null;
+synchronized (this) {
+ if (!unusedEncoders.isEmpty()) {
+encoder = unusedEncoders.remove(0);
+ }
+}
+if (encoder == null) {
+ encoder = new PrefixTreeEncoder(outputStream, includeMvccVersion);
+} else {
+ encoder.reset(outputStream, includeMvccVersion);
+}
+return encoder;
+ }
+
+ @Override
+ public void checkIn(PrefixTreeEncoder encoder) {
+synchronized (this) {
+ this.unusedEncoders.add(encoder);
+}
+ }
+}
{code}
instead of a plain ArrayList for unusedEncoders, maybe you should use a
LinkedBlockingQueue? then you wouldn't have to synchronize the methods
> DBE encode path improvements
>
>
> Key: HBASE-10835
> URL: https://issues.apache.org/jira/browse/HBASE-10835
> Project: HBase
> Issue Type: Improvement
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 0.99.0
>
> Attachments: HBASE-10835.patch, HBASE-10835_V2.patch
>
>
> Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder.
> Encoder again reads kvs one by one from the buffer and encodes and creates a
> new buffer.
> There is no need to have this model now. Previously we had option of no
> encode in disk and encode only in cache. At that time the read buffer from a
> HFile block was passed to this and encodes.
> So encode cell by cell can be done now. Making this change will need us to
> have a NoOp DBE impl which just do the write of a cell as it is with out any
> encoding.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10835) DBE encode path improvements
[ https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13976600#comment-13976600 ] ramkrishna.s.vasudevan commented on HBASE-10835: I have done some. Will do it today or tomorrow again. > DBE encode path improvements > > > Key: HBASE-10835 > URL: https://issues.apache.org/jira/browse/HBASE-10835 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-10835.patch, HBASE-10835_V2.patch > > > Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder. > Encoder again reads kvs one by one from the buffer and encodes and creates a > new buffer. > There is no need to have this model now. Previously we had option of no > encode in disk and encode only in cache. At that time the read buffer from a > HFile block was passed to this and encodes. > So encode cell by cell can be done now. Making this change will need us to > have a NoOp DBE impl which just do the write of a cell as it is with out any > encoding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10835) DBE encode path improvements
[ https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13976593#comment-13976593 ] Anoop Sam John commented on HBASE-10835: Any chance for reviews? > DBE encode path improvements > > > Key: HBASE-10835 > URL: https://issues.apache.org/jira/browse/HBASE-10835 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-10835.patch, HBASE-10835_V2.patch > > > Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder. > Encoder again reads kvs one by one from the buffer and encodes and creates a > new buffer. > There is no need to have this model now. Previously we had option of no > encode in disk and encode only in cache. At that time the read buffer from a > HFile block was passed to this and encodes. > So encode cell by cell can be done now. Making this change will need us to > have a NoOp DBE impl which just do the write of a cell as it is with out any > encoding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10835) DBE encode path improvements
[
https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13969670#comment-13969670
]
Hadoop QA commented on HBASE-10835:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12640277/HBASE-10835_V2.patch
against trunk revision .
ATTACHMENT ID: 12640277
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 18 new
or modified tests.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
at
org.apache.hadoop.hbase.mapreduce.TestImportExport.testImport94Table(TestImportExport.java:230)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9289//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9289//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9289//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9289//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9289//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9289//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9289//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9289//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9289//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9289//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9289//console
This message is automatically generated.
> DBE encode path improvements
>
>
> Key: HBASE-10835
> URL: https://issues.apache.org/jira/browse/HBASE-10835
> Project: HBase
> Issue Type: Improvement
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 0.99.0
>
> Attachments: HBASE-10835.patch, HBASE-10835_V2.patch
>
>
> Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder.
> Encoder again reads kvs one by one from the buffer and encodes and creates a
> new buffer.
> There is no need to have this model now. Previously we had option of no
> encode in disk and encode only in cache. At that time the read buffer from a
> HFile block was passed to this and encodes.
> So encode cell by cell can be done now. Making this change will need us to
> have a NoOp DBE impl which just do the write of a cell as it is with out any
> encoding.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10835) DBE encode path improvements
[
https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13969290#comment-13969290
]
Hadoop QA commented on HBASE-10835:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12640204/HBASE-10835.patch
against trunk revision .
ATTACHMENT ID: 12640204
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 18 new
or modified tests.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+HFileBlockDefaultEncodingContext encodingCtx =
(HFileBlockDefaultEncodingContext) blkEncodingCtx;
+List encodedSeekers = new
ArrayList();
+HFileDataBlockEncoder dbe = (diskAlgo == DataBlockEncoding.NONE) ?
NoOpDataBlockEncoder.INSTANCE
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat2
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.mapreduce.TestMultiTableInputFormat.testScan(TestMultiTableInputFormat.java:244)
at
org.apache.hadoop.hbase.mapreduce.TestMultiTableInputFormat.testScanYZYToEmpty(TestMultiTableInputFormat.java:195)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9288//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9288//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9288//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9288//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9288//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9288//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9288//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9288//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9288//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9288//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9288//console
This message is automatically generated.
> DBE encode path improvements
>
>
> Key: HBASE-10835
> URL: https://issues.apache.org/jira/browse/HBASE-10835
> Project: HBase
> Issue Type: Improvement
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 0.99.0
>
> Attachments: HBASE-10835.patch
>
>
> Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder.
> Encoder again reads kvs one by one from the buffer and encodes and creates a
> new buffer.
> There is no need to have this model now. Previously we had option of no
> encode in disk and encode only in cache. At that time the read buffer from a
> HFile block was passed to this and encodes.
> So encode cell by cell can be done now. Making this change will need us to
> have a NoOp DBE impl which just do the write of a cell as it is with out any
> encoding.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10835) DBE encode path improvements
[ https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13949124#comment-13949124 ] Anoop Sam John commented on HBASE-10835: Thought the change in write path will be bigger and it is not so cell specific so handle in a seperate jira. HBASE-10801 , therea are suggestions on the read path. > DBE encode path improvements > > > Key: HBASE-10835 > URL: https://issues.apache.org/jira/browse/HBASE-10835 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.99.0 > > > Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder. > Encoder again reads kvs one by one from the buffer and encodes and creates a > new buffer. > There is no need to have this model now. Previously we had option of no > encode in disk and encode only in cache. At that time the read buffer from a > HFile block was passed to this and encodes. > So encode cell by cell can be done now. Making this change will need us to > have a NoOp DBE impl which just do the write of a cell as it is with out any > encoding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10835) DBE encode path improvements
[ https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947771#comment-13947771 ] ramkrishna.s.vasudevan commented on HBASE-10835: As said in HBASE-10801 was planning to change the interface of DBEs and address usage of cell. Are you going to change the code without cells? Intention of HBASE-10801 was mainly changing the interface and then doing any improvements on that. (and while doing that I thought the related code also would change may be ending up as what this JIRA suggests). This also includes the read path. > DBE encode path improvements > > > Key: HBASE-10835 > URL: https://issues.apache.org/jira/browse/HBASE-10835 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.99.0 > > > Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder. > Encoder again reads kvs one by one from the buffer and encodes and creates a > new buffer. > There is no need to have this model now. Previously we had option of no > encode in disk and encode only in cache. At that time the read buffer from a > HFile block was passed to this and encodes. > So encode cell by cell can be done now. Making this change will need us to > have a NoOp DBE impl which just do the write of a cell as it is with out any > encoding. -- This message was sent by Atlassian JIRA (v6.2#6252)
