[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14031869#comment-14031869 ] Hudson commented on HBASE-10771: SUCCESS: Integrated in HBase-TRUNK #5208 (See [https://builds.apache.org/job/HBase-TRUNK/5208/]) HBASE-10771 Primitive type put/get APIs in ByteRange (Anoop) (anoopsamjohn: rev 14c2c002966872373312f871f29cee9c1538cd9e) * hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteRange.java * hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestPositionedByteRange.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/SimplePositionedByteRange.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/SimpleByteRange.java * hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestSimpleByteRange.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/PositionedByteRange.java * hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestByteRangeWithKVSerialization.java Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch, HBASE-10771_V3.patch, HBASE-10771_V4.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14031519#comment-14031519 ] Anoop Sam John commented on HBASE-10771: bq.Where do you confirm that the long and int writing is same as BBs Just wrote a test to confirm. I write using BR API and create a BB wrapping the byte[] and do read back using BB APIs. Looks good. Will add that test case also on commit. bq. You should note in the implementations that they are equivalent in a comment on commit. Sure. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch, HBASE-10771_V3.patch, HBASE-10771_V4.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14031697#comment-14031697 ] stack commented on HBASE-10771: --- Still +1 on commit. You can add the notes on commit [~anoop.hbase] Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch, HBASE-10771_V3.patch, HBASE-10771_V4.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018284#comment-14018284 ] stack commented on HBASE-10771: --- +1 Where do you confirm that the long and int writing is same as BBs? You should note in the implementations that they are equivalent in a comment on commit. I just read over this issue. Nice back and forth. Good stuff. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch, HBASE-10771_V3.patch, HBASE-10771_V4.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[
https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008599#comment-14008599
]
Hadoop QA commented on HBASE-10771:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12646734/HBASE-10771_V4.patch
against trunk revision .
ATTACHMENT ID: 12646734
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 3
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 31 release
audit warnings (more than the trunk's current 0 warnings).
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9583//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9583//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9583//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9583//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9583//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9583//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9583//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9583//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9583//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9583//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9583//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9583//console
This message is automatically generated.
Primitive type put/get APIs in ByteRange
-
Key: HBASE-10771
URL: https://issues.apache.org/jira/browse/HBASE-10771
Project: HBase
Issue Type: Improvement
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.99.0
Attachments: HBASE-10771.patch, HBASE-10771_V2.patch,
HBASE-10771_V3.patch, HBASE-10771_V4.patch
While doing HBASE-10713 I came across the need to write int/long (and read
also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add
such APIs.
Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion
under HBASE-10191 suggest we can have BR backed HFileBlocks etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008607#comment-14008607 ] Anoop Sam John commented on HBASE-10771: The javadoc warnings and release audit warnings are not related to this patch Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch, HBASE-10771_V3.patch, HBASE-10771_V4.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13986823#comment-13986823 ] Andrew Purtell commented on HBASE-10771: Visually the get* and put* methods look small enough to be inlineable, but have you confirmed with -XX:+PrintCompilation -XX:+UnlockDiagnosticVMOptions -XX:+PrintInlining ? Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch, HBASE-10771_V3.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13987362#comment-13987362 ] Anoop Sam John commented on HBASE-10771: bq.Might be interesting to add a test to confirm that. Just tested that and it is able to read back using BB APIs. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch, HBASE-10771_V3.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13987363#comment-13987363 ] Anoop Sam John commented on HBASE-10771: bq.Any chance of a unit test that builds and unpacks various structures using the new classes and methods, say 1M times in a loop? Bonus points if it is Cells in HFileBlocks. Will do bq.Visually the get* and put* methods look small enough to be inlineable, but have you confirmed with ... I will check Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch, HBASE-10771_V3.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[
https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13986353#comment-13986353
]
Matt Corgan commented on HBASE-10771:
-
Looks good to me. It looks like the encoding matches the LittleEndian
representation from Bits.java, so a ByteBuffer implementation should be able to
call straight through to its internal putInt method if we deem that's faster
than writing 4 individual bytes to the ByteBuffer.
putInt from java.io.Bits.java{code}
private static byte int3(int x) { return (byte)(x 24); }
private static byte int2(int x) { return (byte)(x 16); }
private static byte int1(int x) { return (byte)(x 8); }
private static byte int0(int x) { return (byte)(x ); }
static void putIntL(ByteBuffer bb, int bi, int x) {
bb._put(bi + 3, int3(x));
bb._put(bi + 2, int2(x));
bb._put(bi + 1, int1(x));
bb._put(bi, int0(x));
}
{code}
putInt from hbase's Bytes.java{code}
public static int putInt(byte[] bytes, int offset, int val) {
if (bytes.length - offset SIZEOF_INT) {
throw new IllegalArgumentException(Not enough room to put an int at
+ offset + offset + in a + bytes.length + byte array);
}
for(int i= offset + 3; i offset; i--) {
bytes[i] = (byte) val;
val = 8;
}
bytes[offset] = (byte) val;
return offset + SIZEOF_INT;
}{code}
Might be interesting to add a test to confirm that. You could write to an
array using a ByteRange and then ByteBuffer.wrap(br.getArray()) and read the
values back.
Primitive type put/get APIs in ByteRange
-
Key: HBASE-10771
URL: https://issues.apache.org/jira/browse/HBASE-10771
Project: HBase
Issue Type: Improvement
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.99.0
Attachments: HBASE-10771.patch, HBASE-10771_V2.patch,
HBASE-10771_V3.patch
While doing HBASE-10713 I came across the need to write int/long (and read
also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add
such APIs.
Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion
under HBASE-10191 suggest we can have BR backed HFileBlocks etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[
https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13978800#comment-13978800
]
Hadoop QA commented on HBASE-10771:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12641506/HBASE-10771_V3.patch
against trunk revision .
ATTACHMENT ID: 12641506
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9375//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9375//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9375//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9375//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9375//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9375//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9375//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9375//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9375//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9375//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9375//console
This message is automatically generated.
Primitive type put/get APIs in ByteRange
-
Key: HBASE-10771
URL: https://issues.apache.org/jira/browse/HBASE-10771
Project: HBase
Issue Type: Improvement
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.99.0
Attachments: HBASE-10771.patch, HBASE-10771_V2.patch,
HBASE-10771_V3.patch
While doing HBASE-10713 I came across the need to write int/long (and read
also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add
such APIs.
Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion
under HBASE-10191 suggest we can have BR backed HFileBlocks etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13979308#comment-13979308 ] Anoop Sam John commented on HBASE-10771: How it looks now [~mcorgan] ? I just removed the hasBytes() stuff etc as of now.. Later when we do more into the offheap work, we can come up with extensions as needed. What do you say [~stack]. Is this patch good to go in? Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch, HBASE-10771_V3.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976601#comment-13976601 ] Anoop Sam John commented on HBASE-10771: bq. maybe it should just return a primitive long and we could have a public static method int numVLongBytes(long n) that quickly computes the number of bytes. Let me try out. bq.does the hbase Bytes util have the exact same format that ByteBuffers use? native java format can be seen in java.nio.Bits.java During writes we compute bytes using HBase Bytes util API. Why BB compare ? Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[
https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977080#comment-13977080
]
Matt Corgan commented on HBASE-10771:
-
{quote}Why BB compare ?{quote}i was thinking for a ByteRange backed by a
ByteBuffer it would be nice if the BR could just forward the getLong(int index)
call to the equivalent underlying BB method. if the formats are different,
then we can't use those methods without a translation layer
Primitive type put/get APIs in ByteRange
-
Key: HBASE-10771
URL: https://issues.apache.org/jira/browse/HBASE-10771
Project: HBase
Issue Type: Improvement
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.99.0
Attachments: HBASE-10771.patch, HBASE-10771_V2.patch
While doing HBASE-10713 I came across the need to write int/long (and read
also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add
such APIs.
Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion
under HBASE-10191 suggest we can have BR backed HFileBlocks etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977812#comment-13977812 ] Anoop Sam John commented on HBASE-10771: A BR impl backed by BB will be another class where the putLong() will be different and same byte representation as in BB. Infact we can call BB.putLong(long). so getLong() can use BB.getLong().So my point is in a byte[] backed current BR impl, we can use HBase Bytes util for the byte[] convertion. Am I missing any thing? Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[
https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965067#comment-13965067
]
Matt Corgan commented on HBASE-10771:
-
taking a look at patch_2 after a mention in HBASE-10861
{code}PairLong, Integer getVLong(int index) throws IOException{code}this
returns a fairly heavyweight object. maybe it should just return a primitive
long and we could have a public static method int numVLongBytes(long n) that
quickly computes the number of bytes. here is a different VLong format i wrote
a while back. i don't know what the numBytes formula for this particular VLong
would be, but you can see how efficiently it could possibly be calculated.
probably much cheaper than creating 3 objects
{code}
public static int numBytes(long in){// do a check for illegal arguments
if not protected
if(in == 0){ return 1; }// doesn't work with the formula below
return (70 - Long.numberOfLeadingZeros(in)) / 7;// 70 comes
from 64+(7-1)
}
public static byte[] getBytes(long value){
int numBytes = numBytes(value);
byte[] bytes = new byte[numBytes];
long remainder = value;
for(int i = 0; i numBytes - 1; ++i){
bytes[i] = (byte)((remainder LONG_7_RIGHT_BITS_SET) |
LONG_8TH_BIT_SET);// set the left bit
remainder = 7;
}
bytes[numBytes - 1] = (byte)(remainder
LONG_7_RIGHT_BITS_SET);// do not set the left bit
return bytes;
}
{code}
also not sure the checked IOException is necessary since all of these methods
could encounter similar corruption errors, plus the actual IO has presumably
already been done earlier.
{code}
+ @Override
+ public long getLong(int index) {
+return Bytes.toLong(bytes, index);
+ }
{code}
does the hbase Bytes util have the exact same format that ByteBuffers use?
native java format can be seen in java.nio.Bits.java
Primitive type put/get APIs in ByteRange
-
Key: HBASE-10771
URL: https://issues.apache.org/jira/browse/HBASE-10771
Project: HBase
Issue Type: Improvement
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.99.0
Attachments: HBASE-10771.patch, HBASE-10771_V2.patch
While doing HBASE-10713 I came across the need to write int/long (and read
also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add
such APIs.
Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion
under HBASE-10191 suggest we can have BR backed HFileBlocks etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955007#comment-13955007 ] Andrew Purtell commented on HBASE-10771: The problems with ByteBuffer as I see it are threefold: 1. Locking and slicing on account of the 'limit' and 'position' fields can be expensive as [~mcorgan] says. 2. The get* and put* methods have bounds checking which add overhead that I think is unnecessary 3. BB methods don't inline well, partially on account of #2. I would not like to see ByteRange be as unsatisfying to use as ByteBuffer. So for me the test as to whether we are doing the right things with ByteRange is that none of the above three points are true for ByteRange. bq. IMO, we're better off making use of an upstream project's types than reinventing our own. We'll enjoy better compatibility with ecosystem tools. Trouble is, ByteBuffer is inflexible and we haven't a concensus on adopting Netty. [~ndimiduk]: Here you repeat my rationale for looking at Netty 4's ByteBuf but I will note that only you seemed to raise an objection to using the Netty 4 ByteBuf classes. This was sufficient to de-prioritize any work on that and so I haven't touched it since our last discussion. I think your suggestion to use reflection will fail to address my point #3 above, as well as be expensive simply due to the use of reflection. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955011#comment-13955011 ] Andrew Purtell commented on HBASE-10771: [~anoop.hbase] and [~ram_krish]: When proposing API additions to ByteRange I think it is important first to establish that no other reasonable alternative exists. Try to change the calling site code to use (Positioned)ByteRange and what it provides today first. The code was written to the ByteBuffer API and may use some BB methods due to its programming model, not actually to serve some essential purpose. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955379#comment-13955379 ] Nick Dimiduk commented on HBASE-10771: -- bq. I will note that only you seemed to raise an objection to using the Netty 4 ByteBuf classes. [~apurtell] I don't recall the specific comment to which you refer. My only objection to Netty4 (or any other alternative, as they are) is a lack of investigation. In no way did I expect such a comment to deter anyone from conducting exactly that investigation! The 3 alternatives I'm aware of are ByteBuffer (perhaps with reflection hacks I experimented with a while back), Netty4 ByteBuf, and roll-our-own ByteRange. My understanding of these options places ByteBuf at the top of my list, but I have only anecdotal evidence to support its use. This is a fundamental component of this system and I'd hope to see the decision driven by data proven through investigation/evaluation as opposed to anecdotal evidence. My primary criteria for a selection would be API compatibility down into HDFS and performance, and really the former is driven by the latter. All of this is just opinion of someone looking on. You three are doing the actual good work here, so please continue doing it. Until such time as I (or anyone else) am submitting patches along with you, please take my concerns as they are: concerns voiced from the peanut-gallery to be considered only if intelligent. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955385#comment-13955385 ] Nick Dimiduk commented on HBASE-10771: -- bq. Can we not resolve these difficulties using subclasses, helper/utility methods, and/or reflection? [~stack] I was referring to subclasses, helper/utility/reflect around ByteBuffers. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955728#comment-13955728 ] stack commented on HBASE-10771: --- Thanks [~ndimiduk] And I like how you put it. My comments are of the same class as yours (to be clear). We have ByteBuffers coming in the front door currently. We give the socket a nioBB to read into. Keeping position in another object, BR for instance seems fine (so we avoid #1 and #2 in @apurtell list above). Reading in from HDFS, we allocate a BB and read into it. Looking at ByteBuf, it has range checking (checkIndex) but the getBytes and the checkIndex are subclasseable so perhaps it could be 'turned off' if we wanted it to be. ByteBuf has nice features like allocations from pools but if we are talking tight loops (no range check) and reuse, then neither nioBB nor nettyBB will do. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[
https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13954825#comment-13954825
]
Matt Corgan commented on HBASE-10771:
-
One of the problems with ByteBuffer is that it's difficult to copy multiple
bytes from BB to byte[] in a multi-threaded scenario, for example, when the
BlockCache is a bunch of DirectBBs and we want to have many simultaneous reader
threads reading from a hot block. For some reason, there's not a method to
copy a range of bytes from a random position in the DBB, so you have to either
acquire a lock on the DBB to take ownership of its position/limit fields, or
you have to slice it to give the reader a copy with its own position/limit.
Both locking and slicing would be very expensive when trying to assemble
encoded Cells in a tight loop.
One solution might be to have a pool of reader threads (i think we already do),
and to give each thread a pre-sliced BB for each big slab in the allocator.
The reader thread could reuse its private copy of the slab BB:
{code}
ByteRange dest = some ByteRange we want to copy into;
//the threadLocalBlock is a DirectByteBufferByteRange in this case.
//allocator sets position/limit to the block boundaries in the underlying slab.
//allocator manages thread-local pool of blocks and does ref counting.
ByteRange threadLocalBlock =
allocator.acquireReadOnlyThreadLocalBlock(blockCacheKey);
dest.copyFrom(threadLocalBlock, srcOffset, destOffset, length);
allocator.releaseReadOnlyThreadLocalBlock(blockCacheKey);
the array-backed ByteRange would do:
class ArrayByteRange implements ByteRange{
void copyFrom(ByteRange src, int srcOffset, int destOffset, int length){
if(src.hasArray()){
System.arraycopy(src.getArray(), srcOffset, this.array, destOffset,
length);
}else{
DirectByteBuffer srcBuffer = ((DirectByteBuffer)src).getBuffer();
int previousPosition = srcBuffer.position();
int previousLimit = srcBuffer.limit();
srcBuffer.position(srcOffset);
srcBuffer.limit(srcOffset + length);
srcBuffer.get(this.array);//copies the bytes
srcBuffer.position(previousPosition);
srcBuffer.limit(previousLimit);
}
}
}
while the DirectByteBufferByteRange would do:
class DirectByteBufferByteRange implements ByteRange{
void copyFrom(ByteRange src, int srcOffset, int destOffset, int length){
if(src.hasArray()){
byte[] srcArray = ((ArrayByteBuffer)src).getArray();
this.buffer.put(srcArray, srcOffset, length);//copies the bytes
}else{
DirectByteBuffer srcBuffer = ((DirectByteBuffer)src).getBuffer();
int previousPosition = srcBuffer.position();
int previousLimit = srcBuffer.limit();
srcBuffer.position(srcOffset);
srcBuffer.limit(srcOffset + length);
this.buffer.put(srcBuffer);//copies the bytes
srcBuffer.position(previousPosition);
srcBuffer.limit(previousLimit);
}
}
}
{code}
Need to be careful with offset/limit calculations. They may be more complex
than in that example.
Primitive type put/get APIs in ByteRange
-
Key: HBASE-10771
URL: https://issues.apache.org/jira/browse/HBASE-10771
Project: HBase
Issue Type: Improvement
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.99.0
Attachments: HBASE-10771.patch, HBASE-10771_V2.patch
While doing HBASE-10713 I came across the need to write int/long (and read
also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add
such APIs.
Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion
under HBASE-10191 suggest we can have BR backed HFileBlocks etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13950599#comment-13950599 ] Anoop Sam John commented on HBASE-10771: What do u say abt above reply [~stack]? Sounds fine? Can I get some +1s pls :) (Latest patch includes more APIs) Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13950613#comment-13950613 ] ramkrishna.s.vasudevan commented on HBASE-10771: +1 on patch. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13951100#comment-13951100 ] stack commented on HBASE-10771: --- Otherwise, the patch seems fine. bq. Adding of this kind of an API will allow some one to have 2 code paths if needed.. Sounds like we will have two code paths in places (smile). Probably unavoidable. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[
https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13951095#comment-13951095
]
stack commented on HBASE-10771:
---
Why again is it that we don't just do ByteBuffer? ByteRange doc says:
{code}
* This interface differs from ByteBuffer:
* liOn-heap bytes only/li
* liRaw {@code byte} access only; does not encode other primitives./li
* liImplements {@code equals(Object)}, {@code #hashCode()}, and
* {@code #compareTo(ByteRange)} so that it can be used in standard java
* Collections. Comparison operations are lexicographic, which is native to
* HBase./li
* liAllows the addition of simple core methods like the deep and shallow
* copy methods./li
* liCan be reused in tight loops like a major compaction which can save
* significant amounts of garbage. (Without reuse, we throw off garbage like
* a href=http://www.youtube.com/watch?v=lkmBH-MjZF4;this thing/a.)/li
* /p
* p
* Mutable, and always evaluates {@code #equals(Object)}, {@code #hashCode()},
* and {@code #compareTo(ByteRange)} based on the current contents.
* /p
* p
* Can contain convenience methods for comparing, printing, cloning, spawning
* new arrays, copying to other arrays, etc. Please place non-core methods into
* {@link ByteRangeUtils}.
{code}
So, we are violating at least the first two items in the Interface with these
changes, right?
Will we want to evolve to this
http://netty.io/4.0/api/io/netty/buffer/ByteBuf.html eventually? Or pull in
some of this functionality too? (If we did ByteBuf, then we'd have other
facility available to us from netty)
Have we written up an end-to-end for ByteRange any place going in and out?
Pardon me if we have and I've just not kept up.
Primitive type put/get APIs in ByteRange
-
Key: HBASE-10771
URL: https://issues.apache.org/jira/browse/HBASE-10771
Project: HBase
Issue Type: Improvement
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.99.0
Attachments: HBASE-10771.patch, HBASE-10771_V2.patch
While doing HBASE-10713 I came across the need to write int/long (and read
also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add
such APIs.
Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion
under HBASE-10191 suggest we can have BR backed HFileBlocks etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13951143#comment-13951143 ] Nick Dimiduk commented on HBASE-10771: -- IMO, we're better off making use of an upstream project's types than reinventing our own. We'll enjoy better compatibility with ecosystem tools. Trouble is, ByteBuffer is inflexible and we haven't a concensus on adopting Netty. The primary concerns of ByteRange are: reusable instances within a tight loop of compactions, simple(r) interface -- something I myself violated in introducing PositionedByteRange, and more relevant comparable implementation. Can we not resolve these difficulties using subclasses, helper/utility methods, and/or reflection? Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13951166#comment-13951166 ] Anoop Sam John commented on HBASE-10771: Oh yes Stack.. We are violating.. My bad I didnt read those javadoc yet. Would like to go with BR instead of BB as we have freedom of going with netty or others. Dont want to make this object fat. That is why was not thinking even adding a mark or reset kind of things also.. Why the put and get are added is it can abstract the impl detail.. Onheap and offheap can impl diff .. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13951730#comment-13951730 ] ramkrishna.s.vasudevan commented on HBASE-10771: Actually I read this javadoc while starting with BR. My bad is that once we wanted this API in BR then we should change the doc. But seeing the above comments I think we should move them to helper classes I think. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13951738#comment-13951738 ] Matt Corgan commented on HBASE-10771: - I'm not sure what the best solution is either. I introduced ByteRange to * wrap cumbersome/fragile array/offset/length logic into a single place for byte[] as String does to char[] * give us native unsigned comparison (it's kinda wild that hbase doesn't have a unified abstraction for this) * add a potentially faster .equals implementation via cached hashCode comparison * make this wrapper object reusable object for highest performance (unlike ByteBuffer) I only intended it to stay simple and am a little worried that creating too many subclasses could hurt its performance in critical paths, but we started talking about using it for more things. I hadn't envisioned adding many methods to the interface, but still thinking on it and having a hard time coming up with negatives. While a hasArray() method could probably be replaced with instanceof and casting by implementations and utilities, it probably wouldn't hurt to put it in the interface. It wasn't the original intention, but ByteRange could be a nice abstraction for moving (or avoiding moving) bytes between ByteBuffers and byte[]'s. Like in the seek method when you want to compare a byte[] key to some bytes in the block cache, the BR implementation can decide whether to do a byte-by-byte comparison behind the scenes (for off-heap blocks) or to use Unsafe to do full word comparisons when both BRs hasArray(), or even copy the off-heap bytes on heap for a comparison. Seems like we should have this layer of abstraction - it gains us really clean code without much of a performance hit. I agree it feels weird to add such basic functionality, but maybe it's just weird that it's not there already. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13951743#comment-13951743 ] stack commented on HBASE-10771: --- [~ndimiduk] What upstream project you thinking? And what we need to adopt netty types? Prove that an async server better than one that starts out w/ hundreds of handlers? Or you mean netty ByteBuf? When you say ...Can we not resolve these difficulties using subclasses, helper/utility methods, and/or reflection?, do you mean subclasses of netty types or ByteBuffer or do you mean subclass of ByteRange? On why BR: bq. ...reusable instances within a tight loop of compactions, simple(r) interface – something I myself violated in introducing PositionedByteRange, and more relevant comparable implementation On being reusable, nioByteBuffer and nettyByteBuf both fail this requirement. What is simple(r) interface? Looks like we are fast undoing this one w/ all the additions being made herein. and ...more relevant comparable implementation prevails. Which we seem to argue proceed with BR... but we need to fix the javadoc up in BR so it aligns w/ these additions. Thanks Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13951747#comment-13951747 ] stack commented on HBASE-10771: --- bq. I agree it feels weird to add such basic functionality, but maybe it's just weird that it's not there already. This is funny. This may actually be the root issue. [~anoop.hbase] You raise this issue because ... in HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. What were the explicit needs? Write timestamp? What else? Where else would we need say a BR#getLong? Do you see the need for it elsewhere in the read/write pipeline? Good on you. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13951750#comment-13951750 ] Anoop Sam John commented on HBASE-10771: Yes Stack.. write and read long is for mvcc. Infact I plan to use put/getVLong(). The current impl what I am testing now uses put/getLong but will change. Also when we write Cell we will need put int/short also Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947760#comment-13947760 ] Anoop Sam John commented on HBASE-10771: Also getBytes() call on an offheap impl should throw Exception rather than copying bytes to onheap and returning. So having an API like hasBytes() will be useful IMHO Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947764#comment-13947764 ] ramkrishna.s.vasudevan commented on HBASE-10771: Yes we need in my opinion. In the patch that i am using i created isDirect() that would say if it is on heap or offheap. hasArray() also is fine. Some of the flows decision making may be based on this. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[
https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13946737#comment-13946737
]
Hadoop QA commented on HBASE-10771:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12636704/HBASE-10771_V2.patch
against trunk revision .
ATTACHMENT ID: 12636704
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testLogRollOnDatanodeDeath(TestLogRolling.java:368)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9090//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9090//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9090//console
This message is automatically generated.
Primitive type put/get APIs in ByteRange
-
Key: HBASE-10771
URL: https://issues.apache.org/jira/browse/HBASE-10771
Project: HBase
Issue Type: Improvement
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.99.0
Attachments: HBASE-10771.patch, HBASE-10771_V2.patch
While doing HBASE-10713 I came across the need to write int/long (and read
also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add
such APIs.
Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion
under HBASE-10191 suggest we can have BR backed HFileBlocks etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13946814#comment-13946814 ] stack commented on HBASE-10771: --- bq. Added API which is equivalent to BB#hasArray(). Is this a good idea? Now we have to have two code paths... one for those with backing byte arrays and then another for those offheap? Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13947544#comment-13947544 ] Anoop Sam John commented on HBASE-10771: bq. Now we have to have two code paths... one for those with backing byte arrays and then another for those offheap? That need not be. Now we have put/get APIs in BR. so the impl can know whether it is on heap or offheap and do the best way of put/get. Another thing can be the comprator. Here also we can come up with a comparator and it can have one code path only which can work with offheap also. Adding of this kind of an API will allow some one to have 2 code paths *if needed*.. As u can see in the path, the put(BR) itself uses this new API. When it is onheap (the src BR) System.arrayCopy can be the best way of copy rather than byte by byte assign (?) When on heap we have no such choice.. So we wont have 2 code paths based on this.. But may be at some use places we *can have also* like above case. (Now or later) How that sounds Stack? Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch, HBASE-10771_V2.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[
https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944779#comment-13944779
]
ramkrishna.s.vasudevan commented on HBASE-10771:
I think we could name the put() as putInt, putShort, putLong and can have the
variant with index as param also.
Also
{code}
+ /**
+ * Store {@code buf} another {@link PositionedByteRange} into the current one
+ * @param buf
+ * the byterange to store.
+ * @return this.
+ */
+ public void put(PositionedByteRange buf);
{code}
{code}
+ /**
+ * Compares the byte retrieved from the {@link PositionedByteRange} with the
specifed byte[] b
+ * @param srcOffset - offset from which the source has to be read for
comparison
+ * @param srcLength - length upto which the source has to be compared
+ * @param b - byte array with comparison is to happen
+ * @param offset - offset of the array to be compared
+ * @param length - length of the array to be compared
+ * @return
+ */
+ public int compare(int srcOffset, int srcLength, byte[] b, int offset, int
length);
+
+ /**
+ * Store {@code buf} another {@link PositionedByteRange} into the current one
+ * @param buf
+ * the byterange to store.
+ * @return this.
+ */
+ public void put(PositionedByteRange buf);
+
+ /**
+ * Copies the bytes from the {@link PositionedByteRange} to the destination
byte array
+ * @param srcOffset - offset from which the copy should happen
+ * @param dest - byte array into which the elements should be copied
+ * @param destOffset - offset of the byte array from which the copy should
happen
+ * @param length - length to be copied
+ * @return
+ */
+ public void copy(int srcOffset, byte[] dest, int destOffset, int length);
{code}
These are things that may be needed. I created other apis like setLimit and
getLimit also to mock the ByteBuffer related things. but that we can add later
also.
Primitive type put/get APIs in ByteRange
-
Key: HBASE-10771
URL: https://issues.apache.org/jira/browse/HBASE-10771
Project: HBase
Issue Type: Improvement
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.99.0
Attachments: HBASE-10771.patch
While doing HBASE-10713 I came across the need to write int/long (and read
also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add
such APIs.
Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion
under HBASE-10191 suggest we can have BR backed HFileBlocks etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944784#comment-13944784 ] ramkrishna.s.vasudevan commented on HBASE-10771: Like we have shallowCopy we may need some ways to actually create a copy of the BR but one that has limits and postion being 'reset' and the other where we just create a copy with the existing limit and position. We may end up in such scenarios. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944787#comment-13944787 ] Anoop Sam John commented on HBASE-10771: bq.I think we could name the put() as putInt, putShort, putLong and can have the variant with index as param also. WIth index as param, APIs are there in BR (in the patch) I thought it will be better to overload the same name with diff arguments. (Like OutputStream write etc). If u strongly feel otherwise I can change. bq.public void put(PositionedByteRange buf); +1 for this. May be better it API can take offset and length also. May be it can be an overloaded one. bq.public int compare(int srcOffset, int srcLength, byte[] b, int offset, int length); Not sure.. I feel these should go to some sort of comparator impl. bq.public void copy(int srcOffset, byte[] dest, int destOffset, int length); We have this get(int index, byte[] dst, int offset, int length) same as copy. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944790#comment-13944790 ] ramkrishna.s.vasudevan commented on HBASE-10771: bq.We have this ,get(int index, byte[] dst, int offset, int length), same as copy. Yes. I did not see that. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944845#comment-13944845 ] ramkrishna.s.vasudevan commented on HBASE-10771: bq. If u strongly feel otherwise I can change. I would prefer putInt, putShort() still. The layer using that can easily know what it is doing. bq.Not sure.. I feel these should go to some sort of comparator impl. May be the name comapre is not right? But it may be needed in place where a given byte[] is checked against the BR contents. For eg in the HfileBlock we compare the HFileBlock's byte[] with a magic array. So in such cases we need this. So it is not necessary a different type of comparator. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13945212#comment-13945212 ] Nick Dimiduk commented on HBASE-10771: -- bq. Encode or decode in place of zero-compressed long values could be helpful OrderedBytes also provides an varint encoding. Have a look at the *Varuint64 methods on that class. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13945794#comment-13945794 ] stack commented on HBASE-10771: --- The hadoop vint code gets bad review http://grepcode.com/file/repo1.maven.org/maven2/org.hbase/asynchbase/1.3.1/org/hbase/async/HBaseRpc.java#818 Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13945850#comment-13945850 ] Andrew Purtell commented on HBASE-10771: bq. I would prefer putInt, putShort() still. The layer using that can easily know what it is doing. +1 bq. The hadoop vint code gets bad review I assume this is a warning about Hadoop vints not an objection to having a vint helper for this class. We are going to want to encode and decode them in DBBs I am positive. I was thinking using of the better versions available recently, like protobuf's, or OrderedBytes. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13945864#comment-13945864 ] stack commented on HBASE-10771: --- vints good.. just not hadoop vints. Copy/paste the pb vints? (They are not accessible as is). Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944405#comment-13944405 ] Anoop Sam John commented on HBASE-10771: That should be good to add Andy. If no objections I can add.. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944092#comment-13944092 ] Anoop Sam John commented on HBASE-10771: Ping [~mcorgan] Any more types support needed Ram? Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944173#comment-13944173 ] Andrew Purtell commented on HBASE-10771: bq. Any more types support needed Since you asked. Encode or decode in place of zero-compressed long values could be helpful. Otherwise we would need to encode into a temporary byte[] and then copy the result. Not sure it would make a difference but can't hurt. Call these getVLong and putVLong. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13944245#comment-13944245 ] Matt Corgan commented on HBASE-10771: - Looks ok to me Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[
https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13943357#comment-13943357
]
Hadoop QA commented on HBASE-10771:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12636051/HBASE-10771.patch
against trunk revision .
ATTACHMENT ID: 12636051
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testLogRollOnDatanodeDeath(TestLogRolling.java:368)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9064//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9064//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9064//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9064//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9064//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9064//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9064//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9064//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9064//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9064//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9064//console
This message is automatically generated.
Primitive type put/get APIs in ByteRange
-
Key: HBASE-10771
URL: https://issues.apache.org/jira/browse/HBASE-10771
Project: HBase
Issue Type: Improvement
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.99.0
Attachments: HBASE-10771.patch
While doing HBASE-10713 I came across the need to write int/long (and read
also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add
such APIs.
Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion
under HBASE-10191 suggest we can have BR backed HFileBlocks etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13943739#comment-13943739 ] Nick Dimiduk commented on HBASE-10771: -- Boilerplate looks good to me ;) +1 Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10771.patch While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13937583#comment-13937583 ] ramkrishna.s.vasudevan commented on HBASE-10771: bq.also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. Yes. Am trying to work on the HFileBlock layer and we found the need for having such getXXX and putXXX type of APIs. Once we have this JIRA we could refactor our code based on this. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13937964#comment-13937964 ] Andrew Purtell commented on HBASE-10771: +1 necessary next step Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13937983#comment-13937983 ] Nick Dimiduk commented on HBASE-10771: -- Correct me if I'm wrong, [~mcorgan], but my understanding of the ByteRange API design was in part to explicitly *not* include all these get/setXXX methods, allowing the focus to be just on the bytes themselves and necessary position state for reading them. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13937989#comment-13937989 ] Andrew Purtell commented on HBASE-10771: Well then you will force us to not use ByteRange for doing anything useful with non-byte[] backed data, right? Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13938001#comment-13938001 ] Andrew Purtell commented on HBASE-10771: I suppose we could create a class ByteRangeUtil and put some static methods there. Or similar. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[ https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13938140#comment-13938140 ] Matt Corgan commented on HBASE-10771: - Nick, I think I was mainly concerned about ByteRanges having a position field, preferring that multiple readers of the ByteRange each tracked their own position. I don't see a problem adding setXXX methods. Just need to clear the cachedHashCode on any modifications. Primitive type put/get APIs in ByteRange - Key: HBASE-10771 URL: https://issues.apache.org/jira/browse/HBASE-10771 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 While doing HBASE-10713 I came across the need to write int/long (and read also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add such APIs. Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion under HBASE-10191 suggest we can have BR backed HFileBlocks etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
