[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15986033#comment-15986033 ] Lars Hofhansl commented on HBASE-17877: --- +1 Unless I hear objections I will commit to branch-1.3, branch-1, and master. (assuming branch-2 is not real) > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Improvement > Components: util >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.branch-1.3.003.patch, HBASE-17877.master.001.patch, > HBASE-17877.master.002.patch, HBASE-17877.master.003.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982367#comment-15982367 ] Vikas Vishwakarma commented on HBASE-17877: --- [~larsh] updated the Guava to v21 and NOTICE.txt file .. credit to Guava :) > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Improvement >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.branch-1.3.003.patch, HBASE-17877.master.001.patch, > HBASE-17877.master.002.patch, HBASE-17877.master.003.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982345#comment-15982345 ] Lars Hofhansl commented on HBASE-17877: --- [~vik.karma] if you have time, sure. Happy to do it too. (credit still all goes to you :) ) > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Improvement >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.master.001.patch, HBASE-17877.master.002.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982327#comment-15982327 ] Vikas Vishwakarma commented on HBASE-17877: --- [~larsh] should I update the patch with the version change and attach new one's ? > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Improvement >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.master.001.patch, HBASE-17877.master.002.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982114#comment-15982114 ] Lars Hofhansl commented on HBASE-17877: --- [~busbey], you are right. We looked at Guava's master branch, but it's like this in v21 as well. Let's change that and the NOTICE file. I can do that and then commit to 1.3.x and following. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Improvement >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.master.001.patch, HBASE-17877.master.002.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[
https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981451#comment-15981451
]
Sean Busbey commented on HBASE-17877:
-
{code}
722 /*
723 * We want to compare only the first index where left[index] !=
right[index]. This
724 * corresponds to the least significant nonzero byte in lw ^
rw, since lw and rw are
725 * little-endian. Long.numberOfTrailingZeros(diff) tells us the
least significant
726 * nonzero bit, and zeroing out the first three bits of L.nTZ
gives us the shift to get
727 * that least significant nonzero byte. This comparison logic
is based on UnsignedBytes
728 * from guava v22.0
729 */
{code}
I don't see a version 22 in the guava repo currently. Could we base this on the
v21 release? or reference a commit hash in their repo if there's a substantial
improvement?
> Replace/improve HBase's byte[] comparator
> -
>
> Key: HBASE-17877
> URL: https://issues.apache.org/jira/browse/HBASE-17877
> Project: HBase
> Issue Type: Improvement
>Reporter: Lars Hofhansl
>Assignee: Vikas Vishwakarma
> Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch,
> 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf,
> HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch,
> HBASE-17877.master.001.patch, HBASE-17877.master.002.patch
>
>
> [~vik.karma] did some extensive tests and found that Hadoop's version is
> faster - dramatically faster in some cases.
> Patch forthcoming.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981449#comment-15981449 ] Sean Busbey commented on HBASE-17877: - If this implementation is a derivative work of the Guava implementation, please note it in NOTICE as we do for other things we've copied from them: [example in master's NOTICE|https://s.apache.org/6mPh] > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Improvement >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.master.001.patch, HBASE-17877.master.002.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981443#comment-15981443 ] Sean Busbey commented on HBASE-17877: - I think this is an improvement rather than a bugfix, and as such should go into the next minor release. So please no 1.2.z at least. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Improvement >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.master.001.patch, HBASE-17877.master.002.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15980678#comment-15980678 ] Duo Zhang commented on HBASE-17877: --- OK. Got it. No other questions. Thanks. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.master.001.patch, HBASE-17877.master.002.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15980674#comment-15980674 ] Vikas Vishwakarma commented on HBASE-17877: --- yes, I was trying to avoid that but it is a fixed cost. The benchmarks without randomizing or having a fixed diff between the arrays was coming out quiet strange and not even varying much with byte array size which it should ideally because as the size increases we will have that many long iterations. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.master.001.patch, HBASE-17877.master.002.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15980667#comment-15980667 ] Duo Zhang commented on HBASE-17877: --- Will the two Random.nextInt calls impact the performance? > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.master.001.patch, HBASE-17877.master.002.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[
https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15980656#comment-15980656
]
Vikas Vishwakarma commented on HBASE-17877:
---
[~Apache9] , I am copying the code below:
{code}
public class MyBenchmark {
static byte[] ba1_8;
static byte[] ba2_8;
...
static {
Random r = new Random();
ba1_8 = new byte[8];
ba2_8 = new byte[8];
r.nextBytes(ba1_8);
r.nextBytes(ba2_8);
}
@GenerateMicroBenchmark
@Fork(1)
@BenchmarkMode(Mode.Throughput)
@Warmup(iterations = 20, time = 1, timeUnit = TimeUnit.SECONDS)
@Measurement(iterations = 50, time = 1, timeUnit = TimeUnit.SECONDS)
public void testHBaseComparator8(BlackHole b) {
// place your benchmarked code here
b.consume(ByteArrayComparator.compareToHbase(ba1_8, 0, ba1_8.length,
ba2_8, 0, ba2_8.length));
}
...
//Similarly for HBase/Hadoop/Guava101 for byte array size from 4 to 16K
int compareToHbase(byte[] buffer1, int offset1, int length1,
byte[] buffer2, int offset2, int length2) {
final int minLength = Math.min(length1, length2);
// Changing input random index using random byte before running
compare
int indx = r.nextInt(minLength);
int cindx = r.nextInt(byteArrLen);
buffer1[indx] = byteArr[cindx];
buffer2[indx] = byteArr[cindx];
//Remaining HBase comparator code
}
}
{code}
> Replace/improve HBase's byte[] comparator
> -
>
> Key: HBASE-17877
> URL: https://issues.apache.org/jira/browse/HBASE-17877
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Vikas Vishwakarma
> Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch,
> 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf,
> HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch,
> HBASE-17877.master.001.patch, HBASE-17877.master.002.patch
>
>
> [~vik.karma] did some extensive tests and found that Hadoop's version is
> faster - dramatically faster in some cases.
> Patch forthcoming.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15980617#comment-15980617 ] Duo Zhang commented on HBASE-17877: --- +1. But I still want to see the final jmh benchmark code. I do not think the jvm optimization could impact the result when you use a static byte array if you implement the benchmark correctly using BlackHole. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.master.001.patch, HBASE-17877.master.002.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15980272#comment-15980272 ] Lars Hofhansl commented on HBASE-17877: --- If there're no further comments... Good to commit? All branches? [~busbey], [~mantonov]? > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.master.001.patch, HBASE-17877.master.002.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979793#comment-15979793 ] Avinash Dongre commented on HBASE-17877: Thanks [~vik.karma] > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.master.001.patch, HBASE-17877.master.002.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979765#comment-15979765 ] Vikas Vishwakarma commented on HBASE-17877: --- [~adongre] I basically used guava UnsignedBytesBenchmark based implementation for the benchmarks running them against different comparator implementations , different byte array sizes, etc .. only difference is to use randomized byte array inputs instead of changing just the last byte else the results are impacted by compiler optimizations https://github.com/google/guava/blob/master/guava-tests/benchmark/com/google/common/primitives/UnsignedBytesBenchmark.java > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.master.001.patch, HBASE-17877.master.002.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979117#comment-15979117 ] Avinash Dongre commented on HBASE-17877: Hi [~vik.karma] Could you please share your JMH bench-marking code ? Thanks > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.master.001.patch, HBASE-17877.master.002.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973578#comment-15973578 ] Steve Loughran commented on HBASE-17877: bq Also while looking at some UnsignedBytes function for the guava version we were using I noticed that guava v14.0.1 also uses this implementation so probably hadoop borrowed it from there .. I was about to say "no it doesn't", but then I saw the comment at the top, "This is borrowed and slightly modified from Guava's". Been in there a long time though (since 2012), so it's history is lost. x86 pert is what really matters, though we don't want to be pathologically antisocial to the other CPU arches, not just for the sake of PPC, but for when Arm goes mainstream in the DC. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.master.001.patch, HBASE-17877.master.002.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973007#comment-15973007 ] stack commented on HBASE-17877: --- +1 > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.master.001.patch, HBASE-17877.master.002.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[
https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972281#comment-15972281
]
Hadoop QA commented on HBASE-17877:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 28s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
68m 7s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 1s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 86m 45s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863749/HBASE-17877.master.002.patch
|
| JIRA Issue | HBASE-17877 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 3f1f34190cee 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / b35121d |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6481/testReport/ |
| modules | C: hbase-common U: hbase-common |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6481/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Replace/improve HBase's byte[] comparator
> -
>
> Key: HBASE-17877
> URL: https://issues.apache.org/jira/browse/HBASE-17877
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Vikas Vishwakarma
> Attachments:
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[
https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972251#comment-15972251
]
Hadoop QA commented on HBASE-17877:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 57s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
54s {color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s
{color} | {color:green} branch-1.3 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s
{color} | {color:green} branch-1.3 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s {color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} branch-1.3 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} branch-1.3 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
15m 12s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 46s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 33s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:66fbe99 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863756/HBASE-17877.branch-1.3.002.patch
|
| JIRA Issue | HBASE-17877 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 3ef35d100065 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
| git
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[
https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972232#comment-15972232
]
Hadoop QA commented on HBASE-17877:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
2s {color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} branch-1.3 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} branch-1.3 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s {color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} branch-1.3 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} branch-1.3 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
17m 7s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 51s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 25s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:66fbe99 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863751/HBASE-17877.branch-1.3.002.patch
|
| JIRA Issue | HBASE-17877 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux c5b76f5f756f 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
| git
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972192#comment-15972192 ] Vikas Vishwakarma commented on HBASE-17877: --- the byte array comparator tests get covered by this unit test TestCellUtil::testMatchingCellFieldsFromByteBufferedCell (matchingRow assertions) > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.master.001.patch, HBASE-17877.master.002.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[
https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972184#comment-15972184
]
Hadoop QA commented on HBASE-17877:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 28s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 9s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
59m 37s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 23s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 53s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863738/HBASE-17877.master.001.patch
|
| JIRA Issue | HBASE-17877 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 70a2db6dd781 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 5eda5fb |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6479/testReport/ |
| modules | C: hbase-common U: hbase-common |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6479/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Replace/improve HBase's byte[] comparator
> -
>
> Key: HBASE-17877
> URL: https://issues.apache.org/jira/browse/HBASE-17877
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Vikas Vishwakarma
> Attachments:
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972138#comment-15972138 ] Vikas Vishwakarma commented on HBASE-17877: --- sure [~larsh] will add a comment in the code and update the patch. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.master.001.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972129#comment-15972129 ] Lars Hofhansl commented on HBASE-17877: --- Should we add a little comment about how most of this code was lifted from Guava? (and perhaps state the version, in case Guava ever changes it's license - unlikely, but whatever) > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.master.001.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972127#comment-15972127 ] Lars Hofhansl commented on HBASE-17877: --- I like -v4! > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.master.001.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[
https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972089#comment-15972089
]
Hadoop QA commented on HBASE-17877:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 34s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 49s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 48s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863727/HBASE-17877.master.001.patch
|
| JIRA Issue | HBASE-17877 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 8cb967236717 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 5eda5fb |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/6477/artifact/patchprocess/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6477/testReport/ |
| modules | C: hbase-common U: hbase-common |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6477/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Replace/improve HBase's byte[] comparator
> -
>
> Key: HBASE-17877
> URL: https://issues.apache.org/jira/browse/HBASE-17877
>
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972057#comment-15972057 ] Vikas Vishwakarma commented on HBASE-17877: --- Ah! I missed to check and thought that the comparator has moved from Bytes to ByteBufferUtils in main branch, updated the main branch patch and also verified 1.3 branch ByteBufferUtils does not need the change [~stack] > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.master.001.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971285#comment-15971285 ] stack commented on HBASE-17877: --- [~vik.karma] Mind combining the change in Bytes and ByteBufferUtils into one patch? Thank you. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.master.001.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970971#comment-15970971 ] Vikas Vishwakarma commented on HBASE-17877: --- [~stack] I have updated the release note, please review the same. I have also logged a jira with Hadoop similar to this one (HADOOP-14313) > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.master.001.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[
https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970895#comment-15970895
]
Hadoop QA commented on HBASE-17877:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 14m 1s
{color} | {color:red} Docker failed to build yetus/hbase:66fbe99. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863626/HBASE-17877.branch-1.3.001.patch
|
| JIRA Issue | HBASE-17877 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6459/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Replace/improve HBase's byte[] comparator
> -
>
> Key: HBASE-17877
> URL: https://issues.apache.org/jira/browse/HBASE-17877
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Vikas Vishwakarma
> Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch,
> 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf,
> HBASE-17877.branch-1.3.001.patch, HBASE-17877.master.001.patch
>
>
> [~vik.karma] did some extensive tests and found that Hadoop's version is
> faster - dramatically faster in some cases.
> Patch forthcoming.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970878#comment-15970878 ] Vikas Vishwakarma commented on HBASE-17877: --- [~anoop.hbase] the issue was a minor bug with my changes in the master branch, now fixed it and uploaded the patch for master branch also. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.master.001.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970711#comment-15970711 ] Vikas Vishwakarma commented on HBASE-17877: --- sure [~anoop.hbase] for main branch I am getting some test failures, will check and update. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970661#comment-15970661 ] Anoop Sam John commented on HBASE-17877: Good.. Can u pls do the similar change in ByteBufferUtils also? > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970469#comment-15970469 ] stack commented on HBASE-17877: --- Beautiful. +1 on commit (guava v4 -int, 0, +int). Suggest you summarize your finding tersely as the release note and that you file a JIRA against HADOOP to push the benefit of your research upstream. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970293#comment-15970293 ] Vikas Vishwakarma commented on HBASE-17877: --- In the above result table guava is v4: -int, 0, +int output range (low performance, but still better than HBase mostly) guava101 is v3: -1, 0, +1 output range (highest performance, much better than hbase/hadoop) > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15970291#comment-15970291 ] Vikas Vishwakarma commented on HBASE-17877: --- thanks for suggesting JMH [~stack] , plan to use it extensively going forward very helpful microbenchmark tool [~larsh] I have uploaded both versions 1. v3 returns -1, 0, +1 range output, same as current HBase comparator 2. v4 returns -int, 0, +int range output same as guava/hadoop but will be different from current HBase comparator output I also did a quick benchmark of the two guava version v3, v4 and it does look like mapping the output in the -1, 0, +1 range has a considerable impact on the microbenchmark. Result table given below. Please review and let me know what you think and based on that I can submit the changes for other branches and also the main branch BBUtil |---|HBase|---|---|guava|---|---|%diff|---|---|guava101|---|---|%diff|---|---| |byte array size|min|mean|max|min|mean|max|min|mean|max|min|mean|max|min|mean|max| |4|20179.995|20239.475|20250.999|24258.298|24286.424|24297.904|20|20|20|23929.66|24021.811|24032.953|19|19|19| |8|20175.597|20228.107|20240.895|22218.243|22251.717|22260.572|10|10|10|22009.894|22042.68|22050.396|9|9|9| |16|19520.124|19589.628|19598.353|22089.079|22111.043|22118.656|13|13|13|21742.71|21780.105|21796.334|11|11|11| |20|18427.668|18489.516|18499.058|18230.004|18556.874|18758.163|-1|0|1|17462.533|17768.048|17808.481|-5|-4|-4| |32|17366.043|17700.693|18987.718|21333.877|21365.919|21374.385|23|21|13|19529.033|19646.884|19659.683|12|11|4| |50|17383.56|17439.878|17519.008|18326.74|18475.674|18688.731|5|6|7|16775.091|16790.448|16798.04|-4|-4|-4| |64|18007.581|18124.243|18138.866|19086.25|21077.556|21127.126|6|16|16|18027.712|18144.897|18158.497|0|0|0| |100|14854.175|14939.755|14957.926|15708.965|15966.999|16002.11|6|7|7|14686.466|14725.467|14741.717|-1|-1|-1| |128|12879.8|14238.741|14406.042|19086.495|19308.204|19335.857|48|36|34|13817.619|14019.693|14682.579|7|-2|2| |200|10595.317|11168.928|11306.439|14495.604|14679.285|14707.823|37|31|30|13701.405|13728.255|13735.985|29|23|21| |256|10054.941|10170.204|10451.762|14563.739|15003.766|15061.606|45|48|44|10902.351|10983.121|11032.754|8|8|6| |512|6543.403|6599.115|6610.697|10325.164|10382.276|10390.722|58|57|57|7361.069|7410.172|7418.975|12|12|12| |1024|3810.905|3827.994|3835.482|7003.569|7025.902|7030.261|84|84|83|4314.714|4325.268|4328.757|13|13|13| |2048|1965.738|2038.953|2042.825|4023.562|4084.072|4088.709|105|100|100|2409.852|2413.554|2416.365|23|18|18| |4096|1060.86|1066.341|1067.135|2021.194|2218.651|2225.33|91|108|109|1294.663|1309.405|1312.426|22|23|23| |8192|545.547|547.181|547.518|1126.356|1135.267|1136.248|106|107|108|678.991|680.187|681.287|24|24|24| |16384|268.177|275.345|275.943|577.081|582.751|584.022|115|112|112|339.917|342.003|342.851|27|24|24| > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967789#comment-15967789 ] stack commented on HBASE-17877: --- (I love this work) > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967397#comment-15967397 ] Vikas Vishwakarma commented on HBASE-17877: --- [~anoop.hbase] will take a look into BBUtils also. [~larsh] one quick doubt. The HBase byte array comparator gives only 3 possible output for any byte array input -1, 0, +1 With Guava and Hadoop comparator the output can be any -integer, 0 , +integer for example , for this input Arrays.fill( ba1_8, (byte) 43 ); Arrays.fill( ba2_8, (byte) 43 ); ba1_8[0] = (byte) 32; ba2_8[0] = (byte) 43; HBase comparator will give output as -1 while Guava & Hadoop comparator will give output as -11 Mostly functions should check comparator output as > < or == , this should be a fair assumption or do we need to do a thorough verification around this? > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965838#comment-15965838 ] Anoop Sam John commented on HBASE-17877: FYI. In master, we have similar compare logic in BBUtils also. And we make use of it.. Need similar correction there as well. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965535#comment-15965535 ] Vikas Vishwakarma commented on HBASE-17877: --- Sure Lars I will put the patch and remove the Guava dependencies also. I adapted the API by adding offset and length as additional parameters. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965441#comment-15965441 ] Lars Hofhansl commented on HBASE-17877: --- Also looking at the code (with offset, as in Hadoop and HBase) I wonder how efficient it is to call unsafe.getLong on an unaligned address (i.e. passing 1-7 as offset). Would it perhaps be better to read the first few bytes (if not aligned) and then switch to 8 bytes processing once we reach an aligned address? > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965426#comment-15965426 ] Lars Hofhansl commented on HBASE-17877: --- Hey [~vik.karma], I notice the Guava version does not allow passing in offset and length. Did you adapt that, or did you test against the API that just takes two byte[]'s? (I'd assume that'd be slightly faster) > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965409#comment-15965409 ] Lars Hofhansl commented on HBASE-17877: --- I like the latest Guava code best anyway. Easier to understand. We can pull that into HBase (but remove Guava dependencies :) ) Want me to do that [~vik.karma] (or wanna do a patch yourself)? > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964156#comment-15964156 ] Vikas Vishwakarma commented on HBASE-17877: --- I completed few iterations today, I added the guava benchmark also which is the latest version, with the following changes 1. Have all the comparable benchmarks in JMH as suggested by [~stack] 2. Added BlackHole.consume for all the benchmark results as suggested by [~Apache9] (thanks again !) 3. Used a slightly optimized random byte generation to make sure it's impact is less on the benchmarks by using a smaller byte array for random byte selection and replacement in input arrays 4. Added the guava benchmarks (master branch) as suggested by [~larsh] above Observations While hadoop version was giving better performance, the performance was ~10% lower when byteArrayLength%8 is not 0, most likely because of the last loop where it iterates over each of the leftover bytes. This also could be some compiler optimization. If I switch the leftover byte handling in hadoop comparator similar to HBase, I get exactly inverse result i.e worse when byteArraySize%8 = 0 and better when byteArraySize%8 != 0 However with guava version I am able to see overall better performance compared to both HBase as well as Hadoop. I had tried the guava version earlier also using native timestamp before/after kind of measurements and in that case hadoop comparator was giving better results in some cases. It could again be statistical variations or related to compiler optimizations, etc. With JMH framework after fixing all the initial issues related to compiler/JIT optimization like input byte array randomization, adding BlackHole, etc I am seeing consistently better benchmarks for the guava version for all array sizes including the one's where byteArraySize%8 is zero or non-zero Looks like the main branch guava version is best performing and replacing that in HBase should give maximum gains (pending review) https://github.com/google/guava/blob/master/guava/src/com/google/common/primitives/UnsignedBytes.java#L362 Results: |---|HBase|---|---|hadoop|---|---|hadoop %diff|---|---|guava|---|---|guava %diff|---|---| |byte array size|min|mean|max|min|mean|max|min|mean|max|min|mean|max|min|mean|max| |4|19814.642|20217.647|20250.91|19838.782|20072.437|20090.503|0|-1|-1|24026.12|24284.021|24300.338|21|20|20| |8|19846.598|19874.477|19881.019|22012.932|22044.713|22051.793|11|11|11|22199.453|22253.173|22261.712|12|12|12| |16|19400.623|19430.837|19438.378|19606.912|19616.322|19649.318|1|1|1|21995.475|22113.443|22120.836|13|14|14| |20|18456.241|18493.416|18500.289|16482.859|16705.744|16776.35|-11|-10|-9|18625.111|18660.355|18704.285|1|1|1| |32|18953.196|18984.412|18992.993|19307.22|19345.122|19352.411|2|2|2|21309.337|21359.051|21377.868|12|13|13| |50|17444.431|17506.91|17518.791|15864.759|15941.543|15953.412|-9|-9|-9|18468.621|18613.202|18749.651|6|6|7| |64|17390.097|18046.898|18143.835|20152.624|20379.32|20397.359|16|13|12|21065.113|21116.799|21128.523|21|17|16| |100|14844.718|14866.353|14889.49|13293.668|13385.7|13403.439|-10|-10|-10|15594.286|15690.369|15796.081|5|6|6| |128|14183.991|14329.948|14351.016|17016.59|17260.48|17278.799|20|20|20|17668.509|19205.199|19333.922|25|34|35| |200|11665.597|11732.09|11748.27|11599.469|11733.228|11755.622|-1|0|0|14540.79|14648.077|14728.363|25|25|25| |256|10404.438|10438.019|10444.734|13205.591|13315.903|13326.772|27|28|28|14448.858|14933.008|15064.242|39|43|44| |512|6405.106|6592.613|6604.371|9031.652|9142.564|9149.54|41|39|39|10236.501|10376.17|10389.971|60|57|57| |1024|3812.341|3832.237|3840.291|3863.105|3864.757|3871.94|1|1|1|6911.951|7002.067|7009.792|81|83|83| |2048|2052.148|2060.585|2061.935|2129.32|2151.807|2155.381|4|4|5|4072.481|4085.278|4089.185|98|98|98| |4096|1073.263|1089.947|1091.566|1069.962|1076.303|1076.993|0|-1|-1|2319.74|2326.514|2328.69|116|113|113| |8192|544.723|547.063|547.449|863.716|866.808|867.296|59|58|58|931.945|1131.406|1136.288|71|107|108| |16384|275.155|275.724|275.909|432.556|434.158|434.698|57|57|58|582.37|584.294|584.852|112|112|112| Apologies for the multiple iterations, I am myself figuring out a lot while doing these microbenchmark iterations and there are multiple dimensions to track in the test at different levels > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962788#comment-15962788 ] Vikas Vishwakarma commented on HBASE-17877: --- Also while looking at some UnsignedBytes function for the guava version we were using I noticed that guava v14.0.1 also uses this implementation so probably hadoop borrowed it from there .. https://github.com/google/guava/blob/v14.0.1/guava/src/com/google/common/primitives/UnsignedBytes.java#L366 > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962785#comment-15962785 ] Vikas Vishwakarma commented on HBASE-17877: --- thanks for pointing that out [~Apache9], I did one more iteration by making the byte 43 also random , that reduces the overall throughput but the trend is similar although downscaled. Also had another issue that the size for the 64 byte array was not set correctly, fixed that too. I haven't used BlackHole to consume the results though .. ||HBase|||Hadoop|||%diff||| |byte array diff index|min|mean|max|min|mean|max|min|mean|max| |4|18942.652|18973.949|18987.352|19763.044|19891.738|19920.7|4|5|5| |8|18442.078|18705.334|18774.54|19239.098|19867.222|19882.932|4|6|6| |16|17947.642|18305.516|18328.247|18689.068|18728.62|18736.506|4|2|2| |20|18131.127|18227.991|18255.977|16191.106|16344.762|16371.75|-11|-10|-10| |32|17836.654|17992.312|18011.39|18104.608|18156.329|18163.441|2|1|1| |50|17092.018|17136.187|17145.662|16188.646|16253.307|16266.726|-5|-5|-5| |64|15619.9|16171.297|16871.696|16931.176|17045.861|17072.514|8|5|1| |100|14598.426|14635.332|14644.213|12997.744|13160.004|13220.913|-11|-10|-10| |128|13787.756|13823.701|13848.585|13971.965|14012.841|14038.761|1|1|1| |200|11489.823|11539.769|11556.094|11354.532|11477.838|11497.657|-1|-1|-1| |256|9488.517|10212.983|10238.57|10237.903|10296.764|10301.186|8|1|1| |512|6611.667|6641.862|6645.826|8535.101|8615.486|8653.864|29|30|30| |1024|3933.345|3939.653|3940.835|4881.189|4965.591|4972.502|24|26|26| |2048|2057.51|2177.584|2191.693|3168.817|3188.902|3192.435|54|46|46| |4096|1124.412|1146.459|1151.985|1617.375|1659.386|1903.191|44|45|65| |8192|510.079|515.797|516.433|555.235|559.029|559.622|9|8|8| |16384|268.525|270.674|271.184|502.553|503.674|504.833|87|86|86| > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962649#comment-15962649 ] Duo Zhang commented on HBASE-17877: --- So finally the arrays will all be filled with byte 43? And have you use BlackHole to consume the compare result? > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962462#comment-15962462 ] Vikas Vishwakarma commented on HBASE-17877: --- Overall there is very good improvement with the hadoop comparator observed using JMH also. There is only the case where the byte array size % 8 is not zero where hadoop comparator is showing slower results. Will check if this can be further optimized by using the HBase handling of the left over bytes after % 8 > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[
https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962461#comment-15962461
]
Vikas Vishwakarma commented on HBASE-17877:
---
[~larsh] here are the updated results.
ok sorry .. figured out a issue with my JMH tests .. but after fixing it the
results are very encouraging. Initially I was using static byte arrays for
comparison which i guess was getting internally optimized.
This was the old JMH benchmark code
{code:title=OldBenchmarkCode.java|borderStyle=solid}
ba1_8 = new byte[8];
ba2_8 = new byte[8];
r.nextBytes(ba1_8);
r.nextBytes(ba2_8);
compareToHadoop(ba1_8, 0, ba1_8.length, ba2_8, 0, ba2_8.length); {
//hadoop comparator code
}
compareToHBase(ba1_8, 0, ba1_8.length, ba2_8, 0, ba2_8.length); {
//hbase comparator code
}
{code}
To avoid any optimizations I changed it as follows by randomly changing one of
the bytes in the byte arrays in the new JMH benchmark code
{code:title=NewBenchmarkCode.java|borderStyle=solid}
ba1_8 = new byte[8];
ba2_8 = new byte[8];
r.nextBytes(ba1_8);
r.nextBytes(ba2_8);
compareToHadoop(ba1_8, 0, ba1_8.length, ba2_8, 0, ba2_8.length); {
final int minLength = Math.min(length1, length2);
int indx = r.nextInt(minLength);
buffer1[indx] = (byte) 43;
buffer2[indx] = (byte) 43;
//hadoop comparator code
}
compareToHBase(ba1_8, 0, ba1_8.length, ba2_8, 0, ba2_8.length); {
final int minLength = Math.min(length1, length2);
int indx = r.nextInt(minLength);
buffer1[indx] = (byte) 43;
buffer2[indx] = (byte) 43;
//hbase comparator code
}
{code}
With the above changes I ran with 20 Warmup cycles and 100 iterations of 1
second each for each array size (so the test duration per comparator is around
30 mins) and now we can clearly see that as the byte array size increases the
throughput in ops/ms reduces and the results are as follows where we see very
good improvement with hadoop comparator vs HBase comparator (except for few
cases).
Iteration#1
||HBase|||Hadoop|||%diff|||
|byte array diff index|min|mean|max|min|mean|max|min|mean|max|
|4|36948.957|37047.507|37063.599|43624.207|43720.104|43736.301|18|18|18|
|8|27884.837|34081.159|34173.034|39546.43|39653.132|39683.029|42|16|16|
|16|32994.729|33606.42|33643.392|38950.12|39033.963|39050.588|18|16|16|
|20|31131.95|31262.936|31427.434|27721.608|27900.273|27934.124|-11|-11|-11|
|32|31564.556|31713.3|31729.588|36641.596|36875.77|36908.993|16|16|16|
|50|25651.127|25704.675|25720.617|21985.286|22783.331|23810.156|-14|-11|-7|
|64|23990.409|25744.616|25817.746|22774.009|22907.009|23040.051|-5|-11|-11|
|100|19559.995|19733.446|19766.259|17116.267|18049.88|19421.504|-12|-9|-2|
|128|20541.274|20564.717|20571.537|27311.353|27444.572|27467.086|33|33|34|
|200|14356.162|14376.86|14384.074|17341.848|17946.231|18587.39|21|25|29|
|256|13319.756|13615.766|13648.414|18262.812|18328.989|18337.549|37|35|34|
|512|8022.747|8053.372|8057.757|12494.631|12560.197|12569.778|56|56|56|
|1024|4368.514|4387.346|4390.766|7049.335|7144.239|7152.564|61|63|63|
|2048|2312.296|2316.975|2318.876|3735.84|3746.904|3748.395|62|62|62|
|4096|963.396|1173.651|1177.635|1854.35|1992.96|1998.702|92|70|70|
|8192|557.483|568.487|568.982|1021.296|1028.422|1029.441|83|81|81|
|16384|270.662|300.638|301.418|512.884|515.227|515.692|89|71|71|
Iteration#2
||HBase|||Hadoop|||%diff|||
|byte array diff index|min|mean|max|min|mean|max|min|mean|max|
|4|35456.243|37025.448|37064.285|43049.677|43680.577|43737.106|21|18|18|
|8|24971.846|33830.522|34169.057|38968.528|39633.455|39725.308|56|17|16|
|16|32733.421|32867.514|32887.865|38875.54|39031.123|39054.413|19|19|19|
|20|29543.281|31638.401|31887.656|27356.015|27902.406|27937.292|-7|-12|-12|
|32|31567.346|31707.575|31730.414|36795.993|36874.896|36905.325|17|16|16|
|50|25178.46|25716.801|25737.88|23123.396|23842.244|23954.188|-8|-7|-7|
|64|25232.908|25769.57|25790.104|23816.636|23926.496|23993.18|-6|-7|-7|
|100|18537.318|19401.317|19450.866|16679.562|17068.408|17110.839|-10|-12|-12|
|128|20516.499|20561.503|20570.657|25857.553|27023.148|27048.004|26|31|31|
|200|14368.006|14387.637|14399.758|16416.011|16620.799|16714.22|14|16|16|
|256|12164.544|13614.59|13646.761|19741.812|19873.067|19881.449|62|46|46|
|512|7949.774|7988.21|8000.377|12369.546|12555.549|12569.001|56|57|57|
|1024|4360.064|4388.154|4391.112|7058.351|7145.961|7152.739|62|63|63|
|2048|2282.581|2315.234|2318.591|3581.612|3741.999|3748.724|57|62|62|
|4096|1129.561|1141.206|1146.345|2013.874|2016.491|2017.712|78|77|76|
|8192|594.281|599.508|600.069|1022.809|1028.663|1029.265|72|72|72|
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962426#comment-15962426 ] Vikas Vishwakarma commented on HBASE-17877: --- [~larsh] updating in 20-30 mins waiting for final iteration to complete .. please ignore the above result. My initial observation with JMH is wrong. I will update the consolidated details in another 1 hour once the test is complete. Basically a issue with the JMH test that i had written and the byte arrays used for comparison was static which probably was leading to some internal optimization. I have fixed that and re-run the tests now I have much better results but doing two iterations to confirm. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962420#comment-15962420 ] Lars Hofhansl commented on HBASE-17877: --- Hi [~vik.karma], none of the scenarios in the table seems 30-40% slower. Is that a different scenario you are referring to in the above text? > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961794#comment-15961794 ] Vikas Vishwakarma commented on HBASE-17877: --- I did some tests with JMH. So overall it looks like the hadoop version gives better throughput with random byte arrays, but it hits a worst case when two the two byte arrays are almost similar and have just 1 byte diff like say last byte in which case it gives 30-40% lower throughput. I will do some more experiments with the guava version also. The below result table is with Random byte arrays. I have used 20 warmup and 20 measurement cycles of 1 sec each. I will leave a longer running test for the weekend. |Byte Array Size|byte array diff index|Mean HBase (ops/ms)|Hadoop (ops/ms) |%diff| |||||| |Random|4|440977.678|439558.547|0| |Random|8|437781.625|439089.09|0| |Random|16|441573.466|439052.733|-1 | |Random|20|433731.957|449067.897|4 | |Random|32|441486.237|448821.457|2 | |Random|50|432720.686|447027.454|3 | |Random|64|436713.574|440450.684|1 | |Random|100|430155.538|439399.306|2 | |Random|128|441413.074|440321.381|0 | |Random|200|450140.537|448164.678|0 | |Random|256|376015.788|457719.989|22 | |Random|512|437394.161|438975.893|0 | |Random|1024|442672.023|440269.773|-1 | |Random|2048|441500.328|439325.611|0 | |Random|4096|428583.365|439881.571|3 | |Random|8192|449889.732|439456.918|-2 | |Random|16384|439970.375|439271.189|0 | > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961633#comment-15961633 ] Lars Hofhansl commented on HBASE-17877: --- OK... Waiting for one more test from [~vik.karma] (updated patch and a few million iteration to get the volatility of the tests down) > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961631#comment-15961631 ] Lars Hofhansl commented on HBASE-17877: --- Just be complete. Here's the plain Gauva version: https://github.com/google/guava/blob/master/guava/src/com/google/common/primitives/UnsignedBytes.java#L362 > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958688#comment-15958688 ] Duo Zhang commented on HBASE-17877: --- [~stack] I think the endian test is for the native platform? X86 is little endian so usually we will use binary search. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958682#comment-15958682 ] Vikas Vishwakarma commented on HBASE-17877: --- that is a good point .. i checked back on this .. In Bytes.java we are using this for the littleEndian check .. this is evaluating to true for me on my dev/prod box static final boolean littleEndian = ByteOrder.nativeOrder().equals(ByteOrder.LITTLE_ENDIAN); Although from what i read online java is supposed to be bigendian .. will check on this also > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958363#comment-15958363 ] stack commented on HBASE-17877: --- bq. The main difference in the two versions is related to the binary search using the diff between the longs being compared in Hadoop version Vs doing Long.reverseBytes and then comparing in HBase version. We only do the binary search if we are NOT bigendian yet java is bigendian? I'm misreading? Thanks. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958354#comment-15958354 ] Vikas Vishwakarma commented on HBASE-17877: --- Sure [~saint@gmail.com] [~Apache9] [~anoopsamjohn] [~lhofhansl], thanks for the inputs Will try to add the micro benchmarks using JMH Will use 1 - 10 million iterations The main difference in the two versions is related to the binary search using the diff between the longs being compared in Hadoop version Vs doing Long.reverseBytes and then comparing in HBase version. For the leftover bytes after all the long compares, in my initial tests I had left it as it is in HBase and then later on changed that to the hadoop implementation which uses guava comparison but there was not much difference. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958302#comment-15958302 ] Duo Zhang commented on HBASE-17877: --- Agree that we'd better use jmh to test the performance. And 1 comparison is too less I think, especially for short key. Thanks. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, > ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[
https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958273#comment-15958273
]
stack commented on HBASE-17877:
---
First, this is great. Thanks for the work [~vik.karma].
Most compares in hbase are in the few byte range so this should help.
jmh is best for this sort of compare going forwrad.
On patch, given attribution ("Stolen from Hadoop..class.. i'd say..."). Speed
up is just in how we handle whats left over after we do the long compares?
> Replace/improve HBase's byte[] comparator
> -
>
> Key: HBASE-17877
> URL: https://issues.apache.org/jira/browse/HBASE-17877
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Vikas Vishwakarma
> Attachments: 17877-1.2.patch, 17877-v2-1.3.patch,
> ByteComparatorJiraHBASE-17877.pdf
>
>
> [~vik.karma] did some extensive tests and found that Hadoop's version is
> faster - dramatically faster in some cases.
> Patch forthcoming.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957181#comment-15957181 ] Lars Hofhansl commented on HBASE-17877: --- We could also flat-out use the Hadoop comparator - or for that matter even the plain Guava comparator. I do think, though, since this is do core to HBase that code should be part of HBase and be changed by us, and not by a project we depend on. Lemme make a new patch bringing back the change [~Apache9] points out - I wonder whether that perhaps is the part that makes it slower in the one case that [~vik.karma] found; and also leave the variables as final. The goal with this was to simply pull the Hadoop code into HBase (and remove the extra Guava dependency). > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956371#comment-15956371 ] Vikas Vishwakarma commented on HBASE-17877: --- [~lhofhansl] I have attached the pdf for the micro benchmarks as well as the YCSB workload benchmarks. Please take a look and let me know if anything more needs to be added. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch, ByteComparatorJiraHBASE-17877.pdf > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[
https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956303#comment-15956303
]
Anoop Sam John commented on HBASE-17877:
Interesting. The main change here is to avoid reverseBytes calls. That is the
thing giving the perf boost?
{code}
if (minLength - offset >= SIZEOF_INT) {
..
1446 offset += SIZEOF_INT;
1447}
1448if (minLength - offset >= SIZEOF_SHORT) {
{code}
These were to avoid byte by byte read and check as much as possible after long
reads. So this is also a needed change?
> Replace/improve HBase's byte[] comparator
> -
>
> Key: HBASE-17877
> URL: https://issues.apache.org/jira/browse/HBASE-17877
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Vikas Vishwakarma
> Attachments: 17877-1.2.patch
>
>
> [~vik.karma] did some extensive tests and found that Hadoop's version is
> faster - dramatically faster in some cases.
> Patch forthcoming.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956274#comment-15956274 ] Lars Hofhansl commented on HBASE-17877: --- But it's certainly a change we can undo and leave it the way it was. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956266#comment-15956266 ] Lars Hofhansl commented on HBASE-17877: --- [~Apache9] that's what the Hadoop comparator does, which tested faster. :) As tests one can use PE and YCSB. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[
https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956166#comment-15956166
]
Duo Zhang commented on HBASE-17877:
---
{noformat}
-int j = minWords << 3; // Same as minWords * SIZEOF_LONG
-for (int i = 0; i < j; i += SIZEOF_LONG) {
+for (int i = 0; i < minWords * SIZEOF_LONG; i += SIZEOF_LONG) {
{noformat}
Why this change?
And mind sharing the code of your perf tests?
Thanks.
> Replace/improve HBase's byte[] comparator
> -
>
> Key: HBASE-17877
> URL: https://issues.apache.org/jira/browse/HBASE-17877
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Vikas Vishwakarma
> Attachments: 17877-1.2.patch
>
>
> [~vik.karma] did some extensive tests and found that Hadoop's version is
> faster - dramatically faster in some cases.
> Patch forthcoming.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956065#comment-15956065 ] Lars Hofhansl commented on HBASE-17877: --- [~vik.karma], can you attach your performance findings? (micro benchmarks and end-to-end) In my own test I use to vet perf changes I found a 20-30% performance improvement - but I am using short key. The effect will be much less pronounced for longer keys (where memory pure bandwidth becomes the dominating factor) > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
