[jira] [Commented] (HBASE-16398) optimize HRegion computeHDFSBlocksDistribution
[ https://issues.apache.org/jira/browse/HBASE-16398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753793#comment-15753793 ] Hudson commented on HBASE-16398: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2140 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2140/]) HBASE-16398 optimize HRegion computeHDFSBlocksDistribution (binlijin: rev 6acbee179feca63d12ea3fff6e9f25a5c044c467) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java > optimize HRegion computeHDFSBlocksDistribution > -- > > Key: HBASE-16398 > URL: https://issues.apache.org/jira/browse/HBASE-16398 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.0, 1.4.0 >Reporter: binlijin >Assignee: binlijin > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16398.branch-1.v1.patch, HBASE-16398.patch, > HBASE-16398_v2.patch, HBASE-16398_v3.patch, HBASE-16398_v4.patch, > HBASE-16398_v5.patch, LocatedBlockStatusComparison.java > > > First i assume there is no reference and link in a region family's directory. > Without the patch to computeHDFSBlocksDistribution for a region family, there > is 1+2*N rpc call, N is hfile numbers, The first rpc call is to > DistributedFileSystem#listStatus to get hfiles, for every hfile there is two > rpc call DistributedFileSystem#getFileStatus(path) and then > DistributedFileSystem#getFileBlockLocations(status, start, length). > With the patch to computeHDFSBlocksDistribution for a region family, there is > 2 rpc call, they are DistributedFileSystem#getFileStatus(path) and > DistributedFileSystem#listLocatedStatus(final Path p, final PathFilter > filter). > So if there is at least one hfile, with the patch, the rpc call will less. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17324) PE Write workload results are wrong
[
https://issues.apache.org/jira/browse/HBASE-17324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753864#comment-15753864
]
Jean-Marc Spaggiari commented on HBASE-17324:
-
Oh! I was not at all doing that for the credit Appy. I was just glad someone
considered this as important.
> PE Write workload results are wrong
> ---
>
> Key: HBASE-17324
> URL: https://issues.apache.org/jira/browse/HBASE-17324
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>
> [~jmspaggi] found this issue of write latencies being unreasonable.
> The reason is, we are using BufferedMutator with size 2MB, so most writes
> return very quick because they get buffered. That's giving us avg. latency
> like 39us. Needs fixing.
> {noformat}
> 16/12/14 12:46:04 INFO hbase.PerformanceEvaluation: Latency (us) :
> mean=39,37, min=2,00, max=27193408,00, stdDev=3443,01, 50th=2,00, 75th=2,00,
> 95th=3,00, 99th=5,00, 99.9th=9157,00, 99.99th=39685,59, 99.999th=160751,28
> 16/12/14 12:46:04 INFO hbase.PerformanceEvaluation: Num measures (latency) :
> 1
> 16/12/14 12:46:04 INFO hbase.PerformanceEvaluation: Mean = 39,37
> Min = 2,00
> Max = 27193408,00
> StdDev= 3443,01
> 50th = 2,00
> 75th = 2,00
> 95th = 3,00
> 99th = 5,00
> 99.9th= 9157,00
> 99.99th = 39685,59
> 99.999th = 160751,28
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17282) Reduce the redundant requests to meta table
[
https://issues.apache.org/jira/browse/HBASE-17282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753884#comment-15753884
]
Hadoop QA commented on HBASE-17282:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 52s
{color} | {color:red} hbase-client in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 49s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 56s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 93m 57s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 137m 57s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843559/HBASE-17282-v1.patch |
| JIRA Issue | HBASE-17282 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 22ee4319ac61 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 6acbee1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4945/artifact/patchprocess/branch-findbugs-hbase-client-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4945/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HB
[jira] [Commented] (HBASE-17324) PE Write workload results are wrong
[
https://issues.apache.org/jira/browse/HBASE-17324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753896#comment-15753896
]
Jingcheng Du commented on HBASE-17324:
--
ycsb has the same problem when testing latency. But it suggests to disable the
client buffer when testing latency.
I guess we can do the same thing in HBase? Enable the buffer when testing
throughput, and disable the buffer in latency.
> PE Write workload results are wrong
> ---
>
> Key: HBASE-17324
> URL: https://issues.apache.org/jira/browse/HBASE-17324
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>
> [~jmspaggi] found this issue of write latencies being unreasonable.
> The reason is, we are using BufferedMutator with size 2MB, so most writes
> return very quick because they get buffered. That's giving us avg. latency
> like 39us. Needs fixing.
> {noformat}
> 16/12/14 12:46:04 INFO hbase.PerformanceEvaluation: Latency (us) :
> mean=39,37, min=2,00, max=27193408,00, stdDev=3443,01, 50th=2,00, 75th=2,00,
> 95th=3,00, 99th=5,00, 99.9th=9157,00, 99.99th=39685,59, 99.999th=160751,28
> 16/12/14 12:46:04 INFO hbase.PerformanceEvaluation: Num measures (latency) :
> 1
> 16/12/14 12:46:04 INFO hbase.PerformanceEvaluation: Mean = 39,37
> Min = 2,00
> Max = 27193408,00
> StdDev= 3443,01
> 50th = 2,00
> 75th = 2,00
> 95th = 3,00
> 99th = 5,00
> 99.9th= 9157,00
> 99.99th = 39685,59
> 99.999th = 160751,28
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17257) Add column-aliasing capability to hbase-client
[
https://issues.apache.org/jira/browse/HBASE-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753905#comment-15753905
]
Hadoop QA commented on HBASE-17257:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 46s
{color} | {color:red} hbase-client in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 51s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 46s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 59s {color}
| {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 105m 0s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 170m 39s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestBufferedMutator |
| | hadoop.hbase.client.TestAsyncGetMultiThread |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843554/HBASE-17257-v4.patch |
| JIRA Issue | HBASE-17257 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux fc5d0be3d913 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 6acbee1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build
[jira] [Commented] (HBASE-15361) Remove unnecessary or Document constraints on BucketCache possible bucket sizes
[
https://issues.apache.org/jira/browse/HBASE-15361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753961#comment-15753961
]
Anoop Sam John commented on HBASE-15361:
HBASE-16993 is trying to solve it.. Having offset state as long only.. Patch
avail there and hopefully can get committed soon.. Am resolving this issue as
a dup then
> Remove unnecessary or Document constraints on BucketCache possible bucket
> sizes
>
>
> Key: HBASE-15361
> URL: https://issues.apache.org/jira/browse/HBASE-15361
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: deepankar
>Priority: Minor
>
> When we were trying to tune the bucket sizes
> {{hbase.bucketcache.bucket.sizes}} according to our workload, we encountered
> an issue due to the way offset is stored in the bucket entry. We divide the
> offset into integer base and byte value and it assumes that all bucket
> offsets will be a multiple of 256 (left shifting by 8). See the code below
> {code}
> long offset() { // Java has no unsigned numbers
> long o = ((long) offsetBase) & 0x;
> o += (((long) (offset1)) & 0xFF) << 32;
> return o << 8;
> }
> private void setOffset(long value) {
> assert (value & 0xFF) == 0;
> value >>= 8;
> offsetBase = (int) value;
> offset1 = (byte) (value >> 32);
> }
> {code}
> This was there to save 3 bytes per BucketEntry instead of using long and when
> there are no other fields in the Bucket Entry, but now there are lot of
> fields in the bucket entry , This not documented so we could either document
> the constraint that it should be a strict 256 bytes multiple of just go away
> with this constraint.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-15361) Remove unnecessary or Document constraints on BucketCache possible bucket sizes
[
https://issues.apache.org/jira/browse/HBASE-15361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John resolved HBASE-15361.
Resolution: Duplicate
> Remove unnecessary or Document constraints on BucketCache possible bucket
> sizes
>
>
> Key: HBASE-15361
> URL: https://issues.apache.org/jira/browse/HBASE-15361
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: deepankar
>Priority: Minor
>
> When we were trying to tune the bucket sizes
> {{hbase.bucketcache.bucket.sizes}} according to our workload, we encountered
> an issue due to the way offset is stored in the bucket entry. We divide the
> offset into integer base and byte value and it assumes that all bucket
> offsets will be a multiple of 256 (left shifting by 8). See the code below
> {code}
> long offset() { // Java has no unsigned numbers
> long o = ((long) offsetBase) & 0x;
> o += (((long) (offset1)) & 0xFF) << 32;
> return o << 8;
> }
> private void setOffset(long value) {
> assert (value & 0xFF) == 0;
> value >>= 8;
> offsetBase = (int) value;
> offset1 = (byte) (value >> 32);
> }
> {code}
> This was there to save 3 bytes per BucketEntry instead of using long and when
> there are no other fields in the Bucket Entry, but now there are lot of
> fields in the bucket entry , This not documented so we could either document
> the constraint that it should be a strict 256 bytes multiple of just go away
> with this constraint.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15248) BLOCKSIZE 4k should result in 4096 bytes on disk; i.e. fit inside a BucketCache 'block' of 4k
[ https://issues.apache.org/jira/browse/HBASE-15248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753974#comment-15753974 ] Anoop Sam John commented on HBASE-15248: BC is having diff sized buckets and slots in it. This sizes as per the possible HFile block sizes. By def we consider 4K, 8K... 512K.. But we give an additional +1KB size for each of the bucket size and so to a slot.. So a 4K is actually becoming 5K. And ya extra bytes are taken off by this cur header and next block header. So right now u have removed the eager fetch of next block header right? [~saint@gmail.com].. Still the cur block, header we will need? Am trying to solve issues under the parent so that I can make HBASE-17204 to happen. > BLOCKSIZE 4k should result in 4096 bytes on disk; i.e. fit inside a > BucketCache 'block' of 4k > - > > Key: HBASE-15248 > URL: https://issues.apache.org/jira/browse/HBASE-15248 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack > > Chatting w/ a gentleman named Daniel Pol who is messing w/ bucketcache, he > wants blocks to be the size specified in the configuration and no bigger. His > hardware set ups fetches pages of 4k and so a block that has 4k of payload > but has then a header and the header of the next block (which helps figure > whats next when scanning) ends up being 4203 bytes or something, and this > then then translates into two seeks per block fetch. > This issue is about what it would take to stay inside our configured size > boundary writing out blocks. > If not possible, give back better signal on what to do so you could fit > inside a particular constraint. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17174) Refactor the AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-17174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-17174: -- Summary: Refactor the AsyncProcess (was: Use shared threadpool and AsyncProcess in BufferedMutatorImpl) > Refactor the AsyncProcess > - > > Key: HBASE-17174 > URL: https://issues.apache.org/jira/browse/HBASE-17174 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17174.v0.patch, HBASE-17174.v1.patch, > HBASE-17174.v10.patch, HBASE-17174.v2.patch, HBASE-17174.v3.patch, > HBASE-17174.v4.patch, HBASE-17174.v5.patch, HBASE-17174.v6.patch, > HBASE-17174.v7.patch, HBASE-17174.v8.patch, HBASE-17174.v9.patch > > > The following are reasons of reusing the pool and AP. > # A update-heavy application, for example, loader, creates many > BufferedMutator for batch updates. But these BufferedMutators can’t share a > large threadpool because the shutdown() method will be called when closing > any BufferedMutator. This patch adds a flag into BufferedMutatorParams for > preventing calling the shutdown() method in BufferedMutatorImpl#close > # The AsyncProcess has the powerful traffic control, but the control is > against the single table currently. Because the AP is constructed at > BufferedMutatorImpl's construction. This patch change the > BufferedMutatorImpl's construction for reuse the AP so that the updates for > different tables can be restricted by the same AP. > Additionally, there are two changes(aren't included in latest patch) for #2. > 1) The AP will be exposed to user. > 2) A new method will be added to Connection for instantiating a AP. > All suggestions are welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16859) Use Bytebuffer pool for non java clients specifically for scans/gets
[ https://issues.apache.org/jira/browse/HBASE-16859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753989#comment-15753989 ] ramkrishna.s.vasudevan commented on HBASE-16859: Ping!!. LAtest patch after review comments updated in the RB. > Use Bytebuffer pool for non java clients specifically for scans/gets > > > Key: HBASE-16859 > URL: https://issues.apache.org/jira/browse/HBASE-16859 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-16859_V1.patch, HBASE-16859_V2.patch, > HBASE-16859_V2.patch, HBASE-16859_V4.patch, HBASE-16859_V5.patch > > > In case of non java clients we still write the results and header into a on > demand byte[]. This can be changed to use the BBPool (onheap or offheap > buffer?). > But the basic problem is to identify if the response is for scans/gets. > - One easy way to do it is use the MethodDescriptor per Call and use the > name of the MethodDescriptor to identify it is a scan/get. But this will > pollute RpcServer by checking for scan/get type response. > - Other way is always set the result to cellScanner but we know that > isClientCellBlockSupported is going to false for non PB clients. So ignore > the cellscanner and go ahead with the results in PB. But this is not clean > - third one is that we already have a RpccallContext being passed to the RS. > In case of scan/gets/multiGets we already set a Rpccallback for shipped call. > So here on response we can check if the callback is not null and check for > isclientBlockSupported. In this case we can get the BB from the pool and > write the result and header to that BB. May be this looks clean? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17174) Refactor the AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-17174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-17174: -- Description: The following are reasons of reusing the pool and AP. # A update-heavy application, for example, loader, creates many BufferedMutator for batch updates. But these BufferedMutators can’t share a large threadpool because the shutdown() method will be called when closing any BufferedMutator. This patch adds a flag into BufferedMutatorParams for preventing calling the shutdown() method in BufferedMutatorImpl#close # The AsyncProcess has the powerful traffic control, but the control is against the single table currently. We should allow alternate traffic control implementation for advanced user who want more control. All suggestions are welcome. was: The following are reasons of reusing the pool and AP. # A update-heavy application, for example, loader, creates many BufferedMutator for batch updates. But these BufferedMutators can’t share a large threadpool because the shutdown() method will be called when closing any BufferedMutator. This patch adds a flag into BufferedMutatorParams for preventing calling the shutdown() method in BufferedMutatorImpl#close # The AsyncProcess has the powerful traffic control, but the control is against the single table currently. Because the AP is constructed at BufferedMutatorImpl's construction. This patch change the BufferedMutatorImpl's construction for reuse the AP so that the updates for different tables can be restricted by the same AP. Additionally, there are two changes(aren't included in latest patch) for #2. 1) The AP will be exposed to user. 2) A new method will be added to Connection for instantiating a AP. All suggestions are welcome. > Refactor the AsyncProcess > - > > Key: HBASE-17174 > URL: https://issues.apache.org/jira/browse/HBASE-17174 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17174.v0.patch, HBASE-17174.v1.patch, > HBASE-17174.v10.patch, HBASE-17174.v2.patch, HBASE-17174.v3.patch, > HBASE-17174.v4.patch, HBASE-17174.v5.patch, HBASE-17174.v6.patch, > HBASE-17174.v7.patch, HBASE-17174.v8.patch, HBASE-17174.v9.patch > > > The following are reasons of reusing the pool and AP. > # A update-heavy application, for example, loader, creates many > BufferedMutator for batch updates. But these BufferedMutators can’t share a > large threadpool because the shutdown() method will be called when closing > any BufferedMutator. This patch adds a flag into BufferedMutatorParams for > preventing calling the shutdown() method in BufferedMutatorImpl#close > # The AsyncProcess has the powerful traffic control, but the control is > against the single table currently. We should allow alternate traffic control > implementation for advanced user who want more control. > All suggestions are welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16398) optimize HRegion computeHDFSBlocksDistribution
[ https://issues.apache.org/jira/browse/HBASE-16398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754006#comment-15754006 ] Hudson commented on HBASE-16398: FAILURE: Integrated in Jenkins build HBase-1.4 #569 (See [https://builds.apache.org/job/HBase-1.4/569/]) HBASE-16398 optimize HRegion computeHDFSBlocksDistribution (binlijin: rev ed393964978969b6f1492a46ddae2893fd2875f4) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java > optimize HRegion computeHDFSBlocksDistribution > -- > > Key: HBASE-16398 > URL: https://issues.apache.org/jira/browse/HBASE-16398 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.0, 1.4.0 >Reporter: binlijin >Assignee: binlijin > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16398.branch-1.v1.patch, HBASE-16398.patch, > HBASE-16398_v2.patch, HBASE-16398_v3.patch, HBASE-16398_v4.patch, > HBASE-16398_v5.patch, LocatedBlockStatusComparison.java > > > First i assume there is no reference and link in a region family's directory. > Without the patch to computeHDFSBlocksDistribution for a region family, there > is 1+2*N rpc call, N is hfile numbers, The first rpc call is to > DistributedFileSystem#listStatus to get hfiles, for every hfile there is two > rpc call DistributedFileSystem#getFileStatus(path) and then > DistributedFileSystem#getFileBlockLocations(status, start, length). > With the patch to computeHDFSBlocksDistribution for a region family, there is > 2 rpc call, they are DistributedFileSystem#getFileStatus(path) and > DistributedFileSystem#listLocatedStatus(final Path p, final PathFilter > filter). > So if there is at least one hfile, with the patch, the rpc call will less. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17314) Limit total buffered size for all replication sources
[
https://issues.apache.org/jira/browse/HBASE-17314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754008#comment-15754008
]
Hadoop QA commented on HBASE-17314:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 51s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 2s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 41s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 4s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 2m
0s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 148m 18s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.replication.multiwal.TestReplicationEndpointWithMultipleAsyncWAL |
| | hadoop.hbase.replication.multiwal.TestReplicationEndpointWithMultipleWAL |
| | hadoop.hbase.replication.TestReplicationEndpoint |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843563/HBASE-17314.v03.patch
|
| JIRA Issue | HBASE-17314 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux dc8fdffc23e6 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 6acbee1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4948/artifact/patchprocess/patch-unit-hbase-serv
[jira] [Commented] (HBASE-17290) Potential loss of data for replication of bulk loaded hfiles
[ https://issues.apache.org/jira/browse/HBASE-17290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754019#comment-15754019 ] Anoop Sam John commented on HBASE-17290: I believe this point was discussed in the original feature issue.. Sorry I totally forgot abt it now.. [~ashish singhi] on vacation right now and once he is back, he can comment abt the discussion points happened at that time. > Potential loss of data for replication of bulk loaded hfiles > > > Key: HBASE-17290 > URL: https://issues.apache.org/jira/browse/HBASE-17290 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu > > Currently the support for replication of bulk loaded hfiles relies on bulk > load marker written in the WAL. > The move of bulk loaded hfile(s) (into region directory) may succeed but the > write of bulk load marker may fail. > This means that although bulk loaded hfile is being served in source cluster, > the replication wouldn't happen. > Normally operator is supposed to retry the bulk load. But relying on human > retry is not robust solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11392) add/remove peer requests should be routed through master
[
https://issues.apache.org/jira/browse/HBASE-11392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754063#comment-15754063
]
Hadoop QA commented on HBASE-11392:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 49s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 21s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 10m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 56s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 50s
{color} | {color:red} hbase-client in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 10m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 15 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 21s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 27s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 57s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 93m 32s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 170m 25s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843564/HBASE-11392-v3.patch |
| JIRA Issue | HBASE-11392 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile cc hbaseprotoc |
| uname | Linux 0fba52ffcd9f 3.13.0-103-generic
[jira] [Updated] (HBASE-17174) Refactor the AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-17174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-17174: -- Attachment: HBASE-17174.v11.patch v11 allows allow alternate traffic control implementation so that we don't expose the AP. > Refactor the AsyncProcess > - > > Key: HBASE-17174 > URL: https://issues.apache.org/jira/browse/HBASE-17174 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17174.v0.patch, HBASE-17174.v1.patch, > HBASE-17174.v10.patch, HBASE-17174.v11.patch, HBASE-17174.v2.patch, > HBASE-17174.v3.patch, HBASE-17174.v4.patch, HBASE-17174.v5.patch, > HBASE-17174.v6.patch, HBASE-17174.v7.patch, HBASE-17174.v8.patch, > HBASE-17174.v9.patch > > > The following are reasons of reusing the pool and AP. > # A update-heavy application, for example, loader, creates many > BufferedMutator for batch updates. But these BufferedMutators can’t share a > large threadpool because the shutdown() method will be called when closing > any BufferedMutator. This patch adds a flag into BufferedMutatorParams for > preventing calling the shutdown() method in BufferedMutatorImpl#close > # The AsyncProcess has the powerful traffic control, but the control is > against the single table currently. We should allow alternate traffic control > implementation for advanced user who want more control. > All suggestions are welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17174) Refactor the AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-17174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-17174: -- Status: Patch Available (was: Open) > Refactor the AsyncProcess > - > > Key: HBASE-17174 > URL: https://issues.apache.org/jira/browse/HBASE-17174 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17174.v0.patch, HBASE-17174.v1.patch, > HBASE-17174.v10.patch, HBASE-17174.v11.patch, HBASE-17174.v2.patch, > HBASE-17174.v3.patch, HBASE-17174.v4.patch, HBASE-17174.v5.patch, > HBASE-17174.v6.patch, HBASE-17174.v7.patch, HBASE-17174.v8.patch, > HBASE-17174.v9.patch > > > The following are reasons of reusing the pool and AP. > # A update-heavy application, for example, loader, creates many > BufferedMutator for batch updates. But these BufferedMutators can’t share a > large threadpool because the shutdown() method will be called when closing > any BufferedMutator. This patch adds a flag into BufferedMutatorParams for > preventing calling the shutdown() method in BufferedMutatorImpl#close > # The AsyncProcess has the powerful traffic control, but the control is > against the single table currently. We should allow alternate traffic control > implementation for advanced user who want more control. > All suggestions are welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17326) Fix findbugs warning in BufferedMutatorParams
Guanghao Zhang created HBASE-17326: -- Summary: Fix findbugs warning in BufferedMutatorParams Key: HBASE-17326 URL: https://issues.apache.org/jira/browse/HBASE-17326 Project: HBase Issue Type: Bug Reporter: Guanghao Zhang https://builds.apache.org/job/PreCommit-HBASE-Build/4947/artifact/patchprocess/branch-findbugs-hbase-client-warnings.html org.apache.hadoop.hbase.client.BufferedMutatorParams defines clone() but doesn't implement Cloneable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17326) Fix findbugs warning in BufferedMutatorParams
[ https://issues.apache.org/jira/browse/HBASE-17326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17326: --- Affects Version/s: 2.0.0 > Fix findbugs warning in BufferedMutatorParams > - > > Key: HBASE-17326 > URL: https://issues.apache.org/jira/browse/HBASE-17326 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang > > https://builds.apache.org/job/PreCommit-HBASE-Build/4947/artifact/patchprocess/branch-findbugs-hbase-client-warnings.html > org.apache.hadoop.hbase.client.BufferedMutatorParams defines clone() but > doesn't implement Cloneable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17254) Do alignment in heapOverhead() for ExtendedCell implementations
[ https://issues.apache.org/jira/browse/HBASE-17254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754267#comment-15754267 ] Xiang Li commented on HBASE-17254: -- Hi [~anoop.hbase], just would like to let you know that I did not stop working on this JIRA but is reading HBASE-15950 and learning MSLAB related JIRA/knowledge > Do alignment in heapOverhead() for ExtendedCell implementations > --- > > Key: HBASE-17254 > URL: https://issues.apache.org/jira/browse/HBASE-17254 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Xiang Li >Assignee: Xiang Li >Priority: Critical > > In KeyValue (and maybe other ExtendedCell implementations), heapOverhead() > does not do alignment(padding gap). I think alignment is required when > calculating heapOverhead, because the space losses due to alignment can not > be used by other objects in heap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17282) Reduce the redundant requests to meta table
[ https://issues.apache.org/jira/browse/HBASE-17282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754380#comment-15754380 ] Duo Zhang commented on HBASE-17282: --- Any concerns? [~carp84] [~stack]. Thanks. > Reduce the redundant requests to meta table > --- > > Key: HBASE-17282 > URL: https://issues.apache.org/jira/browse/HBASE-17282 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17282-v1.patch, HBASE-17282.patch > > > This usually happens at startup when the meta cache is empty. There will be a > lot of locating requests, but most of will have same results. Things become > worse if we do batch operations with AsyncTable as we will send a locating > request for each operation concurrently. > We need to reduce the redundant requests to meta table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17319) Truncate table with preserve after split may cause truncation to fail
[
https://issues.apache.org/jira/browse/HBASE-17319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754655#comment-15754655
]
Ted Yu commented on HBASE-17319:
Please see what may have caused TestTruncateTableProcedure to hang.
For every bug you fix in branch-1, check whether master branch has the same
issue. If so, understand the difference in master and fix the bug as well.
> Truncate table with preserve after split may cause truncation to fail
> -
>
> Key: HBASE-17319
> URL: https://issues.apache.org/jira/browse/HBASE-17319
> Project: HBase
> Issue Type: Bug
> Components: Admin
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Allan Yang
>Assignee: Allan Yang
> Fix For: 1.4.0
>
> Attachments: HBASE-17319-branch-1.patch, HBASE-17319.patch
>
>
> In truncateTableProcedue , when getting tables regions from meta to recreate
> new regions, split parents are not excluded, so the new regions can end up
> with the same start key, and the same region dir:
> {noformat}
> 2016-12-14 20:15:22,231 WARN [RegionOpenAndInitThread-writetest-1]
> regionserver.HRegionFileSystem: Trying to create a region that already exists
> on disk:
> hdfs://hbasedev1/zhengyan-hbase11-func2/.tmp/data/default/writetest/9b2c8d1539cd92661703ceb8a4d518a1
> {noformat}
> The truncateTableProcedue will retry forever and never get success.
> A attached unit test shows everything.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15432) TableInputFormat - support multiple column families scan
[ https://issues.apache.org/jira/browse/HBASE-15432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuesen Liang updated HBASE-15432: - Attachment: HBASE-15432.master.003.patch fix CellCounter > TableInputFormat - support multiple column families scan > > > Key: HBASE-15432 > URL: https://issues.apache.org/jira/browse/HBASE-15432 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0, 0.98.16, 0.98.16.1 >Reporter: nirav patel >Assignee: Xuesen Liang > Fix For: 2.0.0 > > Attachments: HBASE-15432.master.002.patch, > HBASE-15432.master.003.patch, HBASE-15432.master.patch > > > Currently Hbase TableInputFormat class has SCAN_COLUMN_FAMILY and > SCAN_COLUMNS. SCAN_COLUMN_FAMILY can only scan single column family. If we > need to scan multiple column families from a table then we must use > SCAN_COLUMNS where we must provide both columns and column families which is > not a convenient. Can we have a SCAN_COLUMN_FAMILIES which supports scan of > multiple column families. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17254) Do alignment in heapOverhead() for ExtendedCell implementations
[ https://issues.apache.org/jira/browse/HBASE-17254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-17254: --- Priority: Minor (was: Critical) > Do alignment in heapOverhead() for ExtendedCell implementations > --- > > Key: HBASE-17254 > URL: https://issues.apache.org/jira/browse/HBASE-17254 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Xiang Li >Assignee: Xiang Li >Priority: Minor > > In KeyValue (and maybe other ExtendedCell implementations), heapOverhead() > does not do alignment(padding gap). I think alignment is required when > calculating heapOverhead, because the space losses due to alignment can not > be used by other objects in heap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17254) Do alignment in heapOverhead() for ExtendedCell implementations
[ https://issues.apache.org/jira/browse/HBASE-17254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754793#comment-15754793 ] Anoop Sam John commented on HBASE-17254: NP.. Take ur time... BTW, I have changed the issue to be a minor one.. Not so critical any way. > Do alignment in heapOverhead() for ExtendedCell implementations > --- > > Key: HBASE-17254 > URL: https://issues.apache.org/jira/browse/HBASE-17254 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Xiang Li >Assignee: Xiang Li >Priority: Minor > > In KeyValue (and maybe other ExtendedCell implementations), heapOverhead() > does not do alignment(padding gap). I think alignment is required when > calculating heapOverhead, because the space losses due to alignment can not > be used by other objects in heap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17292) Add observer notification before bulk loaded hfile is moved to region directory
[ https://issues.apache.org/jira/browse/HBASE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17292: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Add observer notification before bulk loaded hfile is moved to region > directory > --- > > Key: HBASE-17292 > URL: https://issues.apache.org/jira/browse/HBASE-17292 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 17292.v1.txt, 17292.v2.txt, 17292.v3.txt > > > Currently the postBulkLoadHFile() hook notifies the locations of bulk loaded > hfiles. > However, if bulk load fails after hfile is moved to region directory but > before postBulkLoadHFile() hook is called, there is no way for pluggable > components (replication - see HBASE-17290, backup / restore) to know which > hfile(s) have been moved to region directory. > Even if postBulkLoadHFile() is called in finally block, the write (to backup > table or zookeeper) issued from postBulkLoadHFile() may fail, ending up with > same situation. > This issue adds a preCommitStoreFile() hook which notifies path of to be > committed hfile before bulk loaded hfile is moved to region directory. > With preCommitStoreFile() hook, write (to backup table or zookeeper) can be > issued before the movement of hfile. > If write fails, IOException would make bulk load fail, not leaving hfile in > region directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15432) TableInputFormat - support multiple column families scan
[
https://issues.apache.org/jira/browse/HBASE-15432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755066#comment-15755066
]
Hadoop QA commented on HBASE-15432:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 44s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
1s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 23s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 93m 17s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 133m 35s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843608/HBASE-15432.master.003.patch
|
| JIRA Issue | HBASE-15432 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 4dc8d7593dfe 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 6acbee1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4950/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4950/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> TableInputFormat - support multiple column families scan
>
>
> Key: HBASE-15432
> URL: https://issues.apache.org/jira/browse/HBASE-15432
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 2.0.0, 0.98.16, 0.98.16.1
>Reporter: nirav patel
>Assignee: Xuesen Liang
> Fix For: 2.0.0
>
> Attachments:
[jira] [Commented] (HBASE-15248) BLOCKSIZE 4k should result in 4096 bytes on disk; i.e. fit inside a BucketCache 'block' of 4k
[ https://issues.apache.org/jira/browse/HBASE-15248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755063#comment-15755063 ] stack commented on HBASE-15248: --- bq. So right now u have removed the eager fetch of next block header right? No. I did not. On study, it would be too disruptive removing the fetch of the next block header: HBASE-17185 "Purge the seek of the next block reading HFileBlocks" On this issue, the way we do sizing now -- i.e. we write bigger than what is configured -- is confusing. This issue should be about rationalizing this. > BLOCKSIZE 4k should result in 4096 bytes on disk; i.e. fit inside a > BucketCache 'block' of 4k > - > > Key: HBASE-15248 > URL: https://issues.apache.org/jira/browse/HBASE-15248 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack > > Chatting w/ a gentleman named Daniel Pol who is messing w/ bucketcache, he > wants blocks to be the size specified in the configuration and no bigger. His > hardware set ups fetches pages of 4k and so a block that has 4k of payload > but has then a header and the header of the next block (which helps figure > whats next when scanning) ends up being 4203 bytes or something, and this > then then translates into two seeks per block fetch. > This issue is about what it would take to stay inside our configured size > boundary writing out blocks. > If not possible, give back better signal on what to do so you could fit > inside a particular constraint. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16859) Use Bytebuffer pool for non java clients specifically for scans/gets
[ https://issues.apache.org/jira/browse/HBASE-16859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755097#comment-15755097 ] stack commented on HBASE-16859: --- Any downside [~ram_krish]? > Use Bytebuffer pool for non java clients specifically for scans/gets > > > Key: HBASE-16859 > URL: https://issues.apache.org/jira/browse/HBASE-16859 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-16859_V1.patch, HBASE-16859_V2.patch, > HBASE-16859_V2.patch, HBASE-16859_V4.patch, HBASE-16859_V5.patch > > > In case of non java clients we still write the results and header into a on > demand byte[]. This can be changed to use the BBPool (onheap or offheap > buffer?). > But the basic problem is to identify if the response is for scans/gets. > - One easy way to do it is use the MethodDescriptor per Call and use the > name of the MethodDescriptor to identify it is a scan/get. But this will > pollute RpcServer by checking for scan/get type response. > - Other way is always set the result to cellScanner but we know that > isClientCellBlockSupported is going to false for non PB clients. So ignore > the cellscanner and go ahead with the results in PB. But this is not clean > - third one is that we already have a RpccallContext being passed to the RS. > In case of scan/gets/multiGets we already set a Rpccallback for shipped call. > So here on response we can check if the callback is not null and check for > isclientBlockSupported. In this case we can get the BB from the pool and > write the result and header to that BB. May be this looks clean? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17326) Fix findbugs warning in BufferedMutatorParams
[ https://issues.apache.org/jira/browse/HBASE-17326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17326: -- Attachment: HBASE-17326.master.001.patch > Fix findbugs warning in BufferedMutatorParams > - > > Key: HBASE-17326 > URL: https://issues.apache.org/jira/browse/HBASE-17326 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang > Attachments: HBASE-17326.master.001.patch > > > https://builds.apache.org/job/PreCommit-HBASE-Build/4947/artifact/patchprocess/branch-findbugs-hbase-client-warnings.html > org.apache.hadoop.hbase.client.BufferedMutatorParams defines clone() but > doesn't implement Cloneable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17326) Fix findbugs warning in BufferedMutatorParams
[ https://issues.apache.org/jira/browse/HBASE-17326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17326: -- Assignee: stack Status: Patch Available (was: Open) Fix for two findbugs warnings in master > Fix findbugs warning in BufferedMutatorParams > - > > Key: HBASE-17326 > URL: https://issues.apache.org/jira/browse/HBASE-17326 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Assignee: stack > Attachments: HBASE-17326.master.001.patch > > > https://builds.apache.org/job/PreCommit-HBASE-Build/4947/artifact/patchprocess/branch-findbugs-hbase-client-warnings.html > org.apache.hadoop.hbase.client.BufferedMutatorParams defines clone() but > doesn't implement Cloneable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17149) Procedure v2 - Fix nonce submission
[ https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17149: -- Attachment: HBASE-17149.master.002.patch Retry > Procedure v2 - Fix nonce submission > --- > > Key: HBASE-17149 > URL: https://issues.apache.org/jira/browse/HBASE-17149 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Attachments: HBASE-17149.master.001.patch, > HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, nonce.patch > > > instead of having all the logic in submitProcedure(), split in > registerNonce() + submitProcedure(). > In this case we can avoid calling the coprocessor twice and having a clean > submit logic knowing that there will only be one submission. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17149) Procedure v2 - Fix nonce submission
[ https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755134#comment-15755134 ] stack commented on HBASE-17149: --- Last test run failed parse of TEST-org.apache.hadoop.hbase.client.TestAsyncGetMultiThread.xml.[failed-to-read] > Procedure v2 - Fix nonce submission > --- > > Key: HBASE-17149 > URL: https://issues.apache.org/jira/browse/HBASE-17149 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Attachments: HBASE-17149.master.001.patch, > HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, nonce.patch > > > instead of having all the logic in submitProcedure(), split in > registerNonce() + submitProcedure(). > In this case we can avoid calling the coprocessor twice and having a clean > submit logic knowing that there will only be one submission. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16398) optimize HRegion computeHDFSBlocksDistribution
[ https://issues.apache.org/jira/browse/HBASE-16398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755166#comment-15755166 ] Thiruvel Thirumoolan commented on HBASE-16398: -- Thanks for the patch [~aoxiang]! > optimize HRegion computeHDFSBlocksDistribution > -- > > Key: HBASE-16398 > URL: https://issues.apache.org/jira/browse/HBASE-16398 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.0, 1.4.0 >Reporter: binlijin >Assignee: binlijin > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16398.branch-1.v1.patch, HBASE-16398.patch, > HBASE-16398_v2.patch, HBASE-16398_v3.patch, HBASE-16398_v4.patch, > HBASE-16398_v5.patch, LocatedBlockStatusComparison.java > > > First i assume there is no reference and link in a region family's directory. > Without the patch to computeHDFSBlocksDistribution for a region family, there > is 1+2*N rpc call, N is hfile numbers, The first rpc call is to > DistributedFileSystem#listStatus to get hfiles, for every hfile there is two > rpc call DistributedFileSystem#getFileStatus(path) and then > DistributedFileSystem#getFileBlockLocations(status, start, length). > With the patch to computeHDFSBlocksDistribution for a region family, there is > 2 rpc call, they are DistributedFileSystem#getFileStatus(path) and > DistributedFileSystem#listLocatedStatus(final Path p, final PathFilter > filter). > So if there is at least one hfile, with the patch, the rpc call will less. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17018) Spooling BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755171#comment-15755171 ] stack commented on HBASE-17018: --- Skimmed. Looking clean. Tests look great. > Spooling BufferedMutator > > > Key: HBASE-17018 > URL: https://issues.apache.org/jira/browse/HBASE-17018 > Project: HBase > Issue Type: New Feature >Reporter: Joep Rottinghuis > Attachments: HBASE-17018.master.001.patch, > HBASE-17018.master.002.patch, > HBASE-17018SpoolingBufferedMutatorDesign-v1.pdf, YARN-4061 HBase requirements > for fault tolerant writer.pdf > > > For Yarn Timeline Service v2 we use HBase as a backing store. > A big concern we would like to address is what to do if HBase is > (temporarily) down, for example in case of an HBase upgrade. > Most of the high volume writes will be mostly on a best-effort basis, but > occasionally we do a flush. Mainly during application lifecycle events, > clients will call a flush on the timeline service API. In order to handle the > volume of writes we use a BufferedMutator. When flush gets called on our API, > we in turn call flush on the BufferedMutator. > We would like our interface to HBase be able to spool the mutations to a > filesystems in case of HBase errors. If we use the Hadoop filesystem > interface, this can then be HDFS, gcs, s3, or any other distributed storage. > The mutations can then later be re-played, for example through a MapReduce > job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17148) Procedure v2 - add bulk proc submit
[ https://issues.apache.org/jira/browse/HBASE-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17148: -- Attachment: HBASE-17148.master.002.patch > Procedure v2 - add bulk proc submit > --- > > Key: HBASE-17148 > URL: https://issues.apache.org/jira/browse/HBASE-17148 > Project: HBase > Issue Type: Sub-task > Components: master, proc-v2 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17148-v0.patch, HBASE-17148.master.001.patch, > HBASE-17148.master.002.patch > > > Add the ability to submit multiple procedure as a single operation. useful > for the AM to reduce some lock/unlock/wait times -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17148) Procedure v2 - add bulk proc submit
[ https://issues.apache.org/jira/browse/HBASE-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755254#comment-15755254 ] stack commented on HBASE-17148: --- Fix test breakage. We asserted on count of WALs but had not let the cleaner run. > Procedure v2 - add bulk proc submit > --- > > Key: HBASE-17148 > URL: https://issues.apache.org/jira/browse/HBASE-17148 > Project: HBase > Issue Type: Sub-task > Components: master, proc-v2 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17148-v0.patch, HBASE-17148.master.001.patch, > HBASE-17148.master.002.patch > > > Add the ability to submit multiple procedure as a single operation. useful > for the AM to reduce some lock/unlock/wait times -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17148) Procedure v2 - add bulk proc submit
[ https://issues.apache.org/jira/browse/HBASE-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755256#comment-15755256 ] stack commented on HBASE-17148: --- .002 > Procedure v2 - add bulk proc submit > --- > > Key: HBASE-17148 > URL: https://issues.apache.org/jira/browse/HBASE-17148 > Project: HBase > Issue Type: Sub-task > Components: master, proc-v2 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17148-v0.patch, HBASE-17148.master.001.patch, > HBASE-17148.master.002.patch > > > Add the ability to submit multiple procedure as a single operation. useful > for the AM to reduce some lock/unlock/wait times -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17148) Procedure v2 - add bulk proc submit
[
https://issues.apache.org/jira/browse/HBASE-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755346#comment-15755346
]
Hadoop QA commented on HBASE-17148:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 8s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 16s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 22s
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
7s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 30s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843636/HBASE-17148.master.002.patch
|
| JIRA Issue | HBASE-17148 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 74aa5cc4ab2a 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 6acbee1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4953/testReport/ |
| modules | C: hbase-procedure U: hbase-procedure |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4953/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Procedure v2 - add bulk proc submit
> ---
>
> Key: HBASE-17148
> URL: https://issues.apache.org/jira/browse/HBASE-17148
> Project: HBase
> Issue Type: Sub-task
> Components: master, proc-v2
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17148-v0
[jira] [Updated] (HBASE-17280) Add mechanism to control hbase cleaner behavior
[ https://issues.apache.org/jira/browse/HBASE-17280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Jadhav updated HBASE-17280: Status: Patch Available (was: Open) > Add mechanism to control hbase cleaner behavior > --- > > Key: HBASE-17280 > URL: https://issues.apache.org/jira/browse/HBASE-17280 > Project: HBase > Issue Type: Improvement > Components: Client, hbase, shell >Affects Versions: 1.2.0, 2.0.0 >Reporter: Ajay Jadhav >Priority: Minor > Fix For: 2.0.0, 1.2.0 > > Attachments: HBASE-17280.branch-1.2.patch, > HBASE-17280.branch-2.0.patch > > > Cleaner is used to get rid of archived HFiles and old WALs in HBase. > In the case of heavy workload, cleaner can affect query performance by > creating a lot of connections to perform costly reads/ writes against > underlying filesystem. > This patch allows the user to control HBase cleaner behavior by providing > shell commands to enable/ disable and manually run it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15248) BLOCKSIZE 4k should result in 4096 bytes on disk; i.e. fit inside a BucketCache 'block' of 4k
[ https://issues.apache.org/jira/browse/HBASE-15248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755447#comment-15755447 ] Nick Dimiduk commented on HBASE-15248: -- Yeah we have a disconnect -- BLOCKSIZE doesn't take compression or encoding into account. > BLOCKSIZE 4k should result in 4096 bytes on disk; i.e. fit inside a > BucketCache 'block' of 4k > - > > Key: HBASE-15248 > URL: https://issues.apache.org/jira/browse/HBASE-15248 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack > > Chatting w/ a gentleman named Daniel Pol who is messing w/ bucketcache, he > wants blocks to be the size specified in the configuration and no bigger. His > hardware set ups fetches pages of 4k and so a block that has 4k of payload > but has then a header and the header of the next block (which helps figure > whats next when scanning) ends up being 4203 bytes or something, and this > then then translates into two seeks per block fetch. > This issue is about what it would take to stay inside our configured size > boundary writing out blocks. > If not possible, give back better signal on what to do so you could fit > inside a particular constraint. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17051) libhbase++: implement RPC client and connection management
[ https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HBASE-17051: -- Attachment: HBASE-17051-HBASE-14850.005.patch > libhbase++: implement RPC client and connection management > -- > > Key: HBASE-17051 > URL: https://issues.apache.org/jira/browse/HBASE-17051 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HBASE-17051-HBASE-14850.000.patch, > HBASE-17051-HBASE-14850.001.patch, HBASE-17051-HBASE-14850.002.patch, > HBASE-17051-HBASE-14850.003.patch, HBASE-17051-HBASE-14850.004.patch, > HBASE-17051-HBASE-14850.005.patch > > > This proposes building RPC client and connection management layer, which > supports the equivalent functions resides in RpcClient.java and > RpcConnection.java. Specifically, handler/pipeline concepts will be used for > implementation, similar to NettyRpcClient and NettyRpcConnection in java side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17051) libhbase++: implement RPC client and connection management
[ https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755471#comment-15755471 ] Xiaobing Zhou commented on HBASE-17051: --- Posted v5 to address the last point by adding RpcClient.CreateRpcChannel so that stub can be created by passing RpcChannel. > libhbase++: implement RPC client and connection management > -- > > Key: HBASE-17051 > URL: https://issues.apache.org/jira/browse/HBASE-17051 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HBASE-17051-HBASE-14850.000.patch, > HBASE-17051-HBASE-14850.001.patch, HBASE-17051-HBASE-14850.002.patch, > HBASE-17051-HBASE-14850.003.patch, HBASE-17051-HBASE-14850.004.patch, > HBASE-17051-HBASE-14850.005.patch > > > This proposes building RPC client and connection management layer, which > supports the equivalent functions resides in RpcClient.java and > RpcConnection.java. Specifically, handler/pipeline concepts will be used for > implementation, similar to NettyRpcClient and NettyRpcConnection in java side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17051) libhbase++: implement RPC client and connection management
[
https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755479#comment-15755479
]
Hadoop QA commented on HBASE-17051:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s {color}
| {color:red} HBASE-17051 does not apply to HBASE-14850. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843645/HBASE-17051-HBASE-14850.005.patch
|
| JIRA Issue | HBASE-17051 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4955/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> libhbase++: implement RPC client and connection management
> --
>
> Key: HBASE-17051
> URL: https://issues.apache.org/jira/browse/HBASE-17051
> Project: HBase
> Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HBASE-17051-HBASE-14850.000.patch,
> HBASE-17051-HBASE-14850.001.patch, HBASE-17051-HBASE-14850.002.patch,
> HBASE-17051-HBASE-14850.003.patch, HBASE-17051-HBASE-14850.004.patch,
> HBASE-17051-HBASE-14850.005.patch
>
>
> This proposes building RPC client and connection management layer, which
> supports the equivalent functions resides in RpcClient.java and
> RpcConnection.java. Specifically, handler/pipeline concepts will be used for
> implementation, similar to NettyRpcClient and NettyRpcConnection in java side.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-17269) Intermittent TestMasterProcedureSchedulerConcurrency failure
[
https://issues.apache.org/jira/browse/HBASE-17269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HBASE-17269.
Resolution: Duplicate
> Intermittent TestMasterProcedureSchedulerConcurrency failure
>
>
> Key: HBASE-17269
> URL: https://issues.apache.org/jira/browse/HBASE-17269
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Matteo Bertozzi
>Priority: Minor
>
> TestMasterProcedureSchedulerConcurrency sometimes appeared as timed out test
> in QA runs.
> In
> https://builds.apache.org/job/HBase-TRUNK_matrix/2083/jdk=JDK%201.8%20(latest),label=Hadoop/testReport/org.apache.hadoop.hbase.master.procedure/TestMasterProcedureSchedulerConcurrency/testMasterProcedureSchedulerPerformanceEvaluation/
> :
> I saw:
> {code}
> 2016-12-06 14:22:23,888 ERROR [Time-limited test]
> util.AbstractHBaseTool(151): Error running command-line tool
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1249)
> at java.lang.Thread.join(Thread.java:1323)
> at
> org.apache.hadoop.hbase.master.procedure.MasterProcedureSchedulerPerformanceEvaluation.runThreads(MasterProcedureSchedulerPerformanceEvaluation.java:239)
> at
> org.apache.hadoop.hbase.master.procedure.MasterProcedureSchedulerPerformanceEvaluation.doWork(MasterProcedureSchedulerPerformanceEvaluation.java:261)
> at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:149)
> at
> org.apache.hadoop.hbase.master.procedure.MasterProcedureSchedulerPerformanceEvaluation.main(MasterProcedureSchedulerPerformanceEvaluation.java:294)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17149) Procedure v2 - Fix nonce submission
[
https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755489#comment-15755489
]
Hadoop QA commented on HBASE-17149:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 23 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 11s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 35s
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 109m 25s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 156m 3s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843627/HBASE-17149.master.002.patch
|
| JIRA Issue | HBASE-17149 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux f17a5bc2703d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 6acbee1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4952/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4952/testReport/ |
| modules | C: hbase-procedure hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4952/console |
| Powered by | Ap
[jira] [Commented] (HBASE-17326) Fix findbugs warning in BufferedMutatorParams
[
https://issues.apache.org/jira/browse/HBASE-17326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755491#comment-15755491
]
Hadoop QA commented on HBASE-17326:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 57s
{color} | {color:red} hbase-client in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 21s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 0s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 109m 25s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 156m 37s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843626/HBASE-17326.master.001.patch
|
| JIRA Issue | HBASE-17326 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 49ae4098ef69 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 6acbee1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4951/artifact/patchprocess/branch-findbugs-hbase-client-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4951/artifac
[jira] [Commented] (HBASE-17319) Truncate table with preserve after split may cause truncation to fail
[
https://issues.apache.org/jira/browse/HBASE-17319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755518#comment-15755518
]
Ted Yu commented on HBASE-17319:
In test output for the new test, I found several exceptions in the following
form:
{code}
2016-12-16 20:56:15,107 DEBUG [ProcedureExecutorWorker-1]
procedure2.ProcedureExecutor(1096): rollback attempt failed for
SplitTableRegionProcedure
(table=testTruncateWithPreserveAfterSplit parent region={ENCODED =>
e2a94ed0e554a3d2d4d7c8a283a1f9fd, NAME =>
'testTruncateWithPreserveAfterSplit,c,1481921769384.
e2a94ed0e554a3d2d4d7c8a283a1f9fd.', STARTKEY => 'c', ENDKEY => ''} first
daughter region={ENCODED => 803eb0625a1ee6329963c253ce44a2f3, NAME =>
'testTruncateWithPreserveAfterSplit,c,1481921774661.803eb0625a1ee6329963c253ce44a2f3.',
STARTKEY => 'c', ENDKEY => 'f13:'} and second daughter region={ENCODED =>
60388dba3c28658ac2c3e8a54cb73a7c, NAME =>
'testTruncateWithPreserveAfterSplit,f13:,1481921774661.60388dba3c28658ac2c3e8a54cb73a7c.',
STARTKEY => 'f13:', ENDKEY => ''}) id=7 owner=hbase.hfs.0
state=FINISHED:SPLIT_TABLE_REGION_SET_SPLITTING_TABLE_STATE
failed=java.io.IOException via master-split-region:java.io.IOException: Failed
to update region state to SPLITTING for
testTruncateWithPreserveAfterSplit,c,1481921769384.e2a94ed0e554a3d2d4d7c8a283a1f9fd.
java.io.IOException: Failed to update region state for
testTruncateWithPreserveAfterSplit,c,1481921769384.e2a94ed0e554a3d2d4d7c8a283a1f9fd.
as part of operation for reverting split
at
org.apache.hadoop.hbase.master.procedure.SplitTableRegionProcedure.setRegionStateToRevertSplitting(SplitTableRegionProcedure.java:455)
at
org.apache.hadoop.hbase.master.procedure.SplitTableRegionProcedure.rollbackState(SplitTableRegionProcedure.java:249)
at
org.apache.hadoop.hbase.master.procedure.SplitTableRegionProcedure.rollbackState(SplitTableRegionProcedure.java:73)
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.rollback(StateMachineProcedure.java:175)
at org.apache.hadoop.hbase.procedure2.Procedure.doRollback(Procedure.java:730)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1093)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1049)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:952)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$700(ProcedureExecutor.java:73)
{code}
> Truncate table with preserve after split may cause truncation to fail
> -
>
> Key: HBASE-17319
> URL: https://issues.apache.org/jira/browse/HBASE-17319
> Project: HBase
> Issue Type: Bug
> Components: Admin
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Allan Yang
>Assignee: Allan Yang
> Fix For: 1.4.0
>
> Attachments: HBASE-17319-branch-1.patch, HBASE-17319.patch
>
>
> In truncateTableProcedue , when getting tables regions from meta to recreate
> new regions, split parents are not excluded, so the new regions can end up
> with the same start key, and the same region dir:
> {noformat}
> 2016-12-14 20:15:22,231 WARN [RegionOpenAndInitThread-writetest-1]
> regionserver.HRegionFileSystem: Trying to create a region that already exists
> on disk:
> hdfs://hbasedev1/zhengyan-hbase11-func2/.tmp/data/default/writetest/9b2c8d1539cd92661703ceb8a4d518a1
> {noformat}
> The truncateTableProcedue will retry forever and never get success.
> A attached unit test shows everything.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17148) Procedure v2 - add bulk proc submit
[ https://issues.apache.org/jira/browse/HBASE-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17148: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed Matteo's patch to master branch. > Procedure v2 - add bulk proc submit > --- > > Key: HBASE-17148 > URL: https://issues.apache.org/jira/browse/HBASE-17148 > Project: HBase > Issue Type: Sub-task > Components: master, proc-v2 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17148-v0.patch, HBASE-17148.master.001.patch, > HBASE-17148.master.002.patch > > > Add the ability to submit multiple procedure as a single operation. useful > for the AM to reduce some lock/unlock/wait times -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17149) Procedure v2 - Fix nonce submission
[ https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755582#comment-15755582 ] stack commented on HBASE-17149: --- Failure is HBASE-17323, a common fail on master: > Procedure v2 - Fix nonce submission > --- > > Key: HBASE-17149 > URL: https://issues.apache.org/jira/browse/HBASE-17149 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Attachments: HBASE-17149.master.001.patch, > HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, nonce.patch > > > instead of having all the logic in submitProcedure(), split in > registerNonce() + submitProcedure(). > In this case we can avoid calling the coprocessor twice and having a clean > submit logic knowing that there will only be one submission. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16945) Implement AsyncRegionLocator
[ https://issues.apache.org/jira/browse/HBASE-16945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755584#comment-15755584 ] stack commented on HBASE-16945: --- You seen HBASE-17323 TestAsyncGetMultiThread fails in master [~Apache9]? FYI. > Implement AsyncRegionLocator > > > Key: HBASE-16945 > URL: https://issues.apache.org/jira/browse/HBASE-16945 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16945-v1.patch, HBASE-16945-v2.patch, > HBASE-16945.patch > > > Will implement it based on the small scan introduced in HBASE-16932 and the > async zk cluster registry introduced in HBASE-16835. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17314) Limit total buffered size for all replication sources
[
https://issues.apache.org/jira/browse/HBASE-17314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755588#comment-15755588
]
Ted Yu commented on HBASE-17314:
{code}
936 public static final int REPLICATION_SOURCE_TOTAL_BUFFER_DFAULT = 256
* 1024 * 1024;
{code}
How is the above determined ?
{code}
154 private AtomicInteger bufferUsed;
{code}
Make the variable name more indicative of its purpose. How about naming it
totalBufferUsed ?
{code}
209 this.quotaPermits =
conf.getInt(HConstants.REPLICATION_SOURCE_TOTAL_BUFFER_KEY,
{code}
Align variable name with its purpose.
{code}
130 private AtomicInteger totalBufferUsed = new AtomicInteger();
{code}
Where is the above variable incremented ?
Please fix failing tests.
> Limit total buffered size for all replication sources
> -
>
> Key: HBASE-17314
> URL: https://issues.apache.org/jira/browse/HBASE-17314
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-17314.v01.patch, HBASE-17314.v02.patch,
> HBASE-17314.v03.patch
>
>
> If we have many peers or some servers have many recovered queues, we will
> hold many entries in memory which will increase the pressure of GC, even
> maybe OOM because we will read entries for 64MB to buffer in default for one
> source.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17149) Procedure v2 - Fix nonce submission
[ https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17149: -- Attachment: HBASE-17149.master.002.patch Test fails 50% of the time. Maybe I'll be lucky. > Procedure v2 - Fix nonce submission > --- > > Key: HBASE-17149 > URL: https://issues.apache.org/jira/browse/HBASE-17149 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Attachments: HBASE-17149.master.001.patch, > HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, > HBASE-17149.master.002.patch, nonce.patch > > > instead of having all the logic in submitProcedure(), split in > registerNonce() + submitProcedure(). > In this case we can avoid calling the coprocessor twice and having a clean > submit logic knowing that there will only be one submission. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17326) Fix findbugs warning in BufferedMutatorParams
[ https://issues.apache.org/jira/browse/HBASE-17326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755600#comment-15755600 ] stack commented on HBASE-17326: --- Fail is TestAsyncGetMultiThread.test:146 ? Execution java.lang.NullPointerException which is common issue that fails 50% of the time. See HBASE-17323. Let me commit. > Fix findbugs warning in BufferedMutatorParams > - > > Key: HBASE-17326 > URL: https://issues.apache.org/jira/browse/HBASE-17326 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Assignee: stack > Attachments: HBASE-17326.master.001.patch > > > https://builds.apache.org/job/PreCommit-HBASE-Build/4947/artifact/patchprocess/branch-findbugs-hbase-client-warnings.html > org.apache.hadoop.hbase.client.BufferedMutatorParams defines clone() but > doesn't implement Cloneable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17326) Fix findbugs warning in BufferedMutatorParams
[ https://issues.apache.org/jira/browse/HBASE-17326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17326: -- Resolution: Fixed Fix Version/s: 2.0.0 Status: Resolved (was: Patch Available) > Fix findbugs warning in BufferedMutatorParams > - > > Key: HBASE-17326 > URL: https://issues.apache.org/jira/browse/HBASE-17326 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-17326.master.001.patch > > > https://builds.apache.org/job/PreCommit-HBASE-Build/4947/artifact/patchprocess/branch-findbugs-hbase-client-warnings.html > org.apache.hadoop.hbase.client.BufferedMutatorParams defines clone() but > doesn't implement Cloneable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17149) Procedure v2 - Fix nonce submission
[
https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755602#comment-15755602
]
Hadoop QA commented on HBASE-17149:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s {color}
| {color:red} HBASE-17149 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843653/HBASE-17149.master.002.patch
|
| JIRA Issue | HBASE-17149 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4956/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Procedure v2 - Fix nonce submission
> ---
>
> Key: HBASE-17149
> URL: https://issues.apache.org/jira/browse/HBASE-17149
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Attachments: HBASE-17149.master.001.patch,
> HBASE-17149.master.002.patch, HBASE-17149.master.002.patch,
> HBASE-17149.master.002.patch, nonce.patch
>
>
> instead of having all the logic in submitProcedure(), split in
> registerNonce() + submitProcedure().
> In this case we can avoid calling the coprocessor twice and having a clean
> submit logic knowing that there will only be one submission.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-17327) Allow for lazy connection / BufferedMutator creation
Joep Rottinghuis created HBASE-17327: Summary: Allow for lazy connection / BufferedMutator creation Key: HBASE-17327 URL: https://issues.apache.org/jira/browse/HBASE-17327 Project: HBase Issue Type: Sub-task Reporter: Joep Rottinghuis The SpoolingBufferedMutatorImpl solution (HBASE-17018) solved the use-case when an existing HBase connection stalls out due to HBase being down, or other transient issues. We have a remaining issue that our service will not start up if it cannot initially connect to HBase. This can be solved by letting the SpoolingBufferedMutator create the wrapped BufferedMutatorImpl later, but the problem has already occurred: we already have to have a connection in order to create a BufferedMutator to begin with. It would be good to be able to initiate a connection and then have a way for multiple users to wait for the connection to succeed before using it. I'm thinking perhaps create a LazyConnection interface that extends the Connection interface. It would have an additional waitFor(long timeout, TimeUnit unit) method where clients can wait for the connection to be established before they start using the connection. Or perhaps the ConnectionFactory can have a createLazyConnection method. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17323) TestAsyncGetMultiThread fails in master
[
https://issues.apache.org/jira/browse/HBASE-17323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755676#comment-15755676
]
stack commented on HBASE-17323:
---
Fails 50% of the time:
https://builds.apache.org/job/HBASE-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html
Can repro fail about same rate locally.
Attached patch doesn't fix the issue (though seems to happen less w/ it in
place).
I see this which seems wrong:
{code}
2016-12-16 13:56:41,709 DEBUG [Default-IPC-NioEventLoopGroup-12-16]
client.AsyncRegionLocator(420): The actual exception when updating
region=async,,1481925400233.c6b4c90d218018c1743fb8b6c7b4aee5.,
hostname=kalashnikov.att.net,59461, 1481925396418, seqNum=2
org.apache.hadoop.hbase.CallQueueTooBigException: Call queue is full on
kalashnikov.att.net,59461,1481925396418, too many items queued ?
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:95)
at
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:85)
at
org.apache.hadoop.hbase.client.ConnectionUtils.translateException(ConnectionUtils.java:289)
at
org.apache.hadoop.hbase.client.AsyncSingleRequestRpcRetryingCaller.onError(AsyncSingleRequestRpcRetryingCaller.java:135)
at
org.apache.hadoop.hbase.client.AsyncSingleRequestRpcRetryingCaller.lambda$call$5(AsyncSingleRequestRpcRetryingCaller.java:191)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
at
org.apache.hadoop.hbase.client.RawAsyncTableImpl$1.run(RawAsyncTableImpl.java:123)
at
org.apache.hadoop.hbase.shaded.com.google.protobuf.RpcUtil$1.run(RpcUtil.java:82)
at
org.apache.hadoop.hbase.shaded.com.google.protobuf.RpcUtil$1.run(RpcUtil.java:73)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:389)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:94)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:407)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:403)
at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)
at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)
at
org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:159)
at
org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:189)
{code}
.. so [~Apache9] maybe HBASE-17282 Reduce the redundant requests to meta table
fixes this... let me finish my review there.
> TestAsyncGetMultiThread fails in master
> ---
>
> Key: HBASE-17323
> URL: https://issues.apache.org/jira/browse/HBASE-17323
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
> Attachments: testAsyncGetMultiThread-output.gz
>
>
> From
> https://builds.apache.org/job/HBase-Trunk_matrix/2137/jdk=JDK%201.8%20(latest),label=Hadoop/testReport/junit/org.apache.hadoop.hbase.client/TestAsyncGetMultiThread/test/
> :
> {code}
> java.util.concurrent.ExecutionException: java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.Bytes.toInt(Bytes.java:1003)
> at org.apache.hadoop.hbase.util.Bytes.toInt(Bytes.java:980)
> at
> org.apache.hadoop.hbase.client.TestAsyncGetMultiThread.run(TestAsyncGetMultiThread.java:108)
> at
> org.apache.hadoop.hbase.client.TestAsyncGetMultiThread.lambda$null$1(TestAsyncGetMultiThread.java:122)
> {code}
> This can be reproduced locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17051) libhbase++: implement RPC client and connection management
[
https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755682#comment-15755682
]
Enis Soztutar commented on HBASE-17051:
---
Thanks [~xiaobingo] for the patch.
I'm getting a compilation error, is it only me?
{code}
root@18181e5809da:/usr/src/hbase/hbase-native-client# make
g++ -c connection/connection-pool-test.cc -o
build/debug/connection/connection-pool-test.o -D_GLIBCXX_USE_CXX11_ABI=0 -g
-Wall -std=c++14 -pedantic -fPIC -I. -Ibuild/
g++ -c connection/rpc-client.cc -o build/debug/connection/rpc-client.o
-D_GLIBCXX_USE_CXX11_ABI=0 -g -Wall -std=c++14 -pedantic -fPIC -I. -Ibuild/
In file included from
/usr/include/x86_64-linux-gnu/c++/5/bits/c++allocator.h:33:0,
from /usr/include/c++/5/bits/allocator.h:46,
from /usr/include/c++/5/string:41,
from /usr/local/include/google/protobuf/message.h:114,
from ./connection/request.h:21,
from connection/rpc-client.h:22,
from connection/rpc-client.cc:20:
/usr/include/c++/5/ext/new_allocator.h: In instantiation of 'void
__gnu_cxx::new_allocator<_Tp>::construct(_Up*, _Args&& ...) [with _Up =
google::protobuf::RpcChannel; _Args = {hbase::RpcChannelImplementation*&}; _Tp
= google::protobuf::RpcChannel]':
/usr/include/c++/5/bits/alloc_traits.h:530:4: required from 'static void
std::allocator_traits
>::construct(std::allocator_traits >::allocator_type&,
_Up*, _Args&& ...) [with _Up = google::protobuf::RpcChannel; _Args =
{hbase::RpcChannelImplementation*&}; _Tp = google::protobuf::RpcChannel;
std::allocator_traits >::allocator_type =
std::allocator]'
/usr/include/c++/5/bits/shared_ptr_base.h:522:39: required from
'std::_Sp_counted_ptr_inplace<_Tp, _Alloc,
_Lp>::_Sp_counted_ptr_inplace(_Alloc, _Args&& ...) [with _Args =
{hbase::RpcChannelImplementation*&}; _Tp = google::protobuf::RpcChannel; _Alloc
= std::allocator; __gnu_cxx::_Lock_policy _Lp =
(__gnu_cxx::_Lock_policy)2u]'
/usr/include/c++/5/bits/shared_ptr_base.h:617:4: required from
'std::__shared_count<_Lp>::__shared_count(std::_Sp_make_shared_tag, _Tp*, const
_Alloc&, _Args&& ...) [with _Tp = google::protobuf::RpcChannel; _Alloc =
std::allocator; _Args =
{hbase::RpcChannelImplementation*&}; __gnu_cxx::_Lock_policy _Lp =
(__gnu_cxx::_Lock_policy)2u]'
/usr/include/c++/5/bits/shared_ptr_base.h:1097:35: required from
'std::__shared_ptr<_Tp, _Lp>::__shared_ptr(std::_Sp_make_shared_tag, const
_Alloc&, _Args&& ...) [with _Alloc =
std::allocator; _Args =
{hbase::RpcChannelImplementation*&}; _Tp = google::protobuf::RpcChannel;
__gnu_cxx::_Lock_policy _Lp = (__gnu_cxx::_Lock_policy)2u]'
/usr/include/c++/5/bits/shared_ptr.h:319:64: required from
'std::shared_ptr<_Tp>::shared_ptr(std::_Sp_make_shared_tag, const _Alloc&,
_Args&& ...) [with _Alloc = std::allocator; _Args
= {hbase::RpcChannelImplementation*&}; _Tp = google::protobuf::RpcChannel]'
/usr/include/c++/5/bits/shared_ptr.h:620:39: required from
'std::shared_ptr<_Tp1> std::allocate_shared(const _Alloc&, _Args&& ...) [with
_Tp = google::protobuf::RpcChannel; _Alloc =
std::allocator; _Args =
{hbase::RpcChannelImplementation*&}]'
/usr/include/c++/5/bits/shared_ptr.h:635:39: required from
'std::shared_ptr<_Tp1> std::make_shared(_Args&& ...) [with _Tp =
google::protobuf::RpcChannel; _Args = {hbase::RpcChannelImplementation*&}]'
connection/rpc-client.cc:120:46: required from here
/usr/include/c++/5/ext/new_allocator.h:120:4: error: invalid new-expression of
abstract class type 'google::protobuf::RpcChannel'
{ ::new((void *)__p) _Up(std::forward<_Args>(__args)...); }
^
In file included from connection/rpc-client.h:28:0,
from connection/rpc-client.cc:20:
/usr/local/include/google/protobuf/service.h:269:26: note: because the
following virtual functions are pure within 'google::protobuf::RpcChannel':
class LIBPROTOBUF_EXPORT RpcChannel {
^
/usr/local/include/google/protobuf/service.h:279:16: note: virtual void
google::protobuf::RpcChannel::CallMethod(const
google::protobuf::MethodDescriptor*, google::protobuf::RpcController*, const
google::protobuf::Message*, google::protobuf::Message*,
google::protobuf::Closure*)
virtual void CallMethod(const MethodDescriptor* method,
^
Makefile:112: recipe for target 'build/debug/connection/rpc-client.o' failed
{code}
> libhbase++: implement RPC client and connection management
> --
>
> Key: HBASE-17051
> URL: https://issues.apache.org/jira/browse/HBASE-17051
> Project: HBase
> Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HBASE-17051-HBASE-14850.000.patch,
> HBASE-17051-HBASE-14850.001.patch, HBASE-1705
[jira] [Updated] (HBASE-17323) TestAsyncGetMultiThread fails in master
[
https://issues.apache.org/jira/browse/HBASE-17323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-17323:
--
Attachment: 17323.hack.txt
Code that checks for null and if we are stopping, just skips out. Also started
to refactor the call queue too big exception to add more info which is nice but
this not the locale of the exception pasted above... more to do.
> TestAsyncGetMultiThread fails in master
> ---
>
> Key: HBASE-17323
> URL: https://issues.apache.org/jira/browse/HBASE-17323
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
> Attachments: 17323.hack.txt, testAsyncGetMultiThread-output.gz
>
>
> From
> https://builds.apache.org/job/HBase-Trunk_matrix/2137/jdk=JDK%201.8%20(latest),label=Hadoop/testReport/junit/org.apache.hadoop.hbase.client/TestAsyncGetMultiThread/test/
> :
> {code}
> java.util.concurrent.ExecutionException: java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.Bytes.toInt(Bytes.java:1003)
> at org.apache.hadoop.hbase.util.Bytes.toInt(Bytes.java:980)
> at
> org.apache.hadoop.hbase.client.TestAsyncGetMultiThread.run(TestAsyncGetMultiThread.java:108)
> at
> org.apache.hadoop.hbase.client.TestAsyncGetMultiThread.lambda$null$1(TestAsyncGetMultiThread.java:122)
> {code}
> This can be reproduced locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17280) Add mechanism to control hbase cleaner behavior
[
https://issues.apache.org/jira/browse/HBASE-17280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755707#comment-15755707
]
Hadoop QA commented on HBASE-17280:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 33s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 1s
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 1s
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 4m 57s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
7s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s
{color} | {color:green} branch-1.2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s
{color} | {color:green} branch-1.2 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m
5s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
0s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
59s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} branch-1.2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} branch-1.2 passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 8 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 13s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 17s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 35s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 73m 53s
{color} | {color:green} hbase-server
[jira] [Commented] (HBASE-17051) libhbase++: implement RPC client and connection management
[
https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755708#comment-15755708
]
Enis Soztutar commented on HBASE-17051:
---
Nvm, I think this:
{code}
std::shared_ptr RpcClient::CreateRpcChannel(const std::string &host,
uint16_t port, std::shared_ptr ticket, int rpc_timeout) {
auto channel = new RpcChannelImplementation(shared_from_this(), host, port,
ticket, rpc_timeout);
return std::make_shared(channel);
}
{code}
should be:
{code}
std::shared_ptr RpcClient::CreateRpcChannel(const std::string &host,
uint16_t port, std::shared_ptr ticket, int rpc_timeout) {
return std::make_shared(shared_from_this(),
host, port, ticket, rpc_timeout);
}
{code}
Let me attach a patch.
> libhbase++: implement RPC client and connection management
> --
>
> Key: HBASE-17051
> URL: https://issues.apache.org/jira/browse/HBASE-17051
> Project: HBase
> Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HBASE-17051-HBASE-14850.000.patch,
> HBASE-17051-HBASE-14850.001.patch, HBASE-17051-HBASE-14850.002.patch,
> HBASE-17051-HBASE-14850.003.patch, HBASE-17051-HBASE-14850.004.patch,
> HBASE-17051-HBASE-14850.005.patch
>
>
> This proposes building RPC client and connection management layer, which
> supports the equivalent functions resides in RpcClient.java and
> RpcConnection.java. Specifically, handler/pipeline concepts will be used for
> implementation, similar to NettyRpcClient and NettyRpcConnection in java side.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17323) TestAsyncGetMultiThread fails in master
[
https://issues.apache.org/jira/browse/HBASE-17323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755716#comment-15755716
]
stack commented on HBASE-17323:
---
I still see above exception even with HBASE-17282 in place. HBASE-17282 does
not fix this issue either.
> TestAsyncGetMultiThread fails in master
> ---
>
> Key: HBASE-17323
> URL: https://issues.apache.org/jira/browse/HBASE-17323
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
> Attachments: 17323.hack.txt, testAsyncGetMultiThread-output.gz
>
>
> From
> https://builds.apache.org/job/HBase-Trunk_matrix/2137/jdk=JDK%201.8%20(latest),label=Hadoop/testReport/junit/org.apache.hadoop.hbase.client/TestAsyncGetMultiThread/test/
> :
> {code}
> java.util.concurrent.ExecutionException: java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.Bytes.toInt(Bytes.java:1003)
> at org.apache.hadoop.hbase.util.Bytes.toInt(Bytes.java:980)
> at
> org.apache.hadoop.hbase.client.TestAsyncGetMultiThread.run(TestAsyncGetMultiThread.java:108)
> at
> org.apache.hadoop.hbase.client.TestAsyncGetMultiThread.lambda$null$1(TestAsyncGetMultiThread.java:122)
> {code}
> This can be reproduced locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-17328) Properly dispose of looped replication peers
Vincent Poon created HBASE-17328:
Summary: Properly dispose of looped replication peers
Key: HBASE-17328
URL: https://issues.apache.org/jira/browse/HBASE-17328
Project: HBase
Issue Type: Bug
Components: Replication
Affects Versions: 0.98.23, 2.0.0, 1.4.0
Reporter: Vincent Poon
When adding a looped replication peer (clusterId == peerClusterId), the
following code terminates the replication source thread, but since the source
manager still holds a reference, WALs continue to get enqueued, and never get
cleaned because they're stuck in the queue, leading to an unsustainable
buildup. Furthermore, the replication statistics thread will continue to print
statistics for the terminated source.
{code}
if (clusterId.equals(peerClusterId) &&
!replicationEndpoint.canReplicateToSameCluster()) {
this.terminate("ClusterId " + clusterId + " is replicating to itself:
peerClusterId "
+ peerClusterId + " which is not allowed by ReplicationEndpoint:"
+ replicationEndpoint.getClass().getName(), null, false);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-17328) Properly dispose of looped replication peers
[
https://issues.apache.org/jira/browse/HBASE-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vincent Poon reassigned HBASE-17328:
Assignee: Vincent Poon
> Properly dispose of looped replication peers
>
>
> Key: HBASE-17328
> URL: https://issues.apache.org/jira/browse/HBASE-17328
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.4.0, 0.98.23
>Reporter: Vincent Poon
>Assignee: Vincent Poon
>
> When adding a looped replication peer (clusterId == peerClusterId), the
> following code terminates the replication source thread, but since the source
> manager still holds a reference, WALs continue to get enqueued, and never get
> cleaned because they're stuck in the queue, leading to an unsustainable
> buildup. Furthermore, the replication statistics thread will continue to
> print statistics for the terminated source.
> {code}
> if (clusterId.equals(peerClusterId) &&
> !replicationEndpoint.canReplicateToSameCluster()) {
> this.terminate("ClusterId " + clusterId + " is replicating to itself:
> peerClusterId "
> + peerClusterId + " which is not allowed by ReplicationEndpoint:"
> + replicationEndpoint.getClass().getName(), null, false);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17051) [C++] implement RPC client and connection management
[ https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-17051: -- Summary: [C++] implement RPC client and connection management (was: libhbase++: implement RPC client and connection management) > [C++] implement RPC client and connection management > > > Key: HBASE-17051 > URL: https://issues.apache.org/jira/browse/HBASE-17051 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HBASE-17051-HBASE-14850.000.patch, > HBASE-17051-HBASE-14850.001.patch, HBASE-17051-HBASE-14850.002.patch, > HBASE-17051-HBASE-14850.003.patch, HBASE-17051-HBASE-14850.004.patch, > HBASE-17051-HBASE-14850.005.patch > > > This proposes building RPC client and connection management layer, which > supports the equivalent functions resides in RpcClient.java and > RpcConnection.java. Specifically, handler/pipeline concepts will be used for > implementation, similar to NettyRpcClient and NettyRpcConnection in java side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17292) Add observer notification before bulk loaded hfile is moved to region directory
[ https://issues.apache.org/jira/browse/HBASE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17292: --- Attachment: 17292.addendum Addendum fixes a bug where the path to copied hfile is not returned by preBulkLoadHFile(). > Add observer notification before bulk loaded hfile is moved to region > directory > --- > > Key: HBASE-17292 > URL: https://issues.apache.org/jira/browse/HBASE-17292 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 17292.addendum, 17292.v1.txt, 17292.v2.txt, 17292.v3.txt > > > Currently the postBulkLoadHFile() hook notifies the locations of bulk loaded > hfiles. > However, if bulk load fails after hfile is moved to region directory but > before postBulkLoadHFile() hook is called, there is no way for pluggable > components (replication - see HBASE-17290, backup / restore) to know which > hfile(s) have been moved to region directory. > Even if postBulkLoadHFile() is called in finally block, the write (to backup > table or zookeeper) issued from postBulkLoadHFile() may fail, ending up with > same situation. > This issue adds a preCommitStoreFile() hook which notifies path of to be > committed hfile before bulk loaded hfile is moved to region directory. > With preCommitStoreFile() hook, write (to backup table or zookeeper) can be > issued before the movement of hfile. > If write fails, IOException would make bulk load fail, not leaving hfile in > region directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17148) Procedure v2 - add bulk proc submit
[ https://issues.apache.org/jira/browse/HBASE-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755816#comment-15755816 ] Hudson commented on HBASE-17148: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2144 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2144/]) HBASE-17148 Procedure v2 - add bulk proc submit (Matteo Bertozzi) (stack: rev e16e2a61c143ce95c632b92d38f8d95815b08fdf) * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/ProcedureStore.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java * (edit) hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/TestProcedureExecutor.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/NoopProcedureStore.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/Procedure.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/ProcedureStoreTracker.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java * (edit) hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/wal/TestWALProcedureStore.java > Procedure v2 - add bulk proc submit > --- > > Key: HBASE-17148 > URL: https://issues.apache.org/jira/browse/HBASE-17148 > Project: HBase > Issue Type: Sub-task > Components: master, proc-v2 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17148-v0.patch, HBASE-17148.master.001.patch, > HBASE-17148.master.002.patch > > > Add the ability to submit multiple procedure as a single operation. useful > for the AM to reduce some lock/unlock/wait times -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17329) Improve message when CallQueueTooBigException
stack created HBASE-17329: - Summary: Improve message when CallQueueTooBigException Key: HBASE-17329 URL: https://issues.apache.org/jira/browse/HBASE-17329 Project: HBase Issue Type: Improvement Reporter: stack Assignee: stack Priority: Trivial Print out queue item counts and queue sizes. While here correct queue name spellings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17329) Improve message when CallQueueTooBigException
[ https://issues.apache.org/jira/browse/HBASE-17329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17329: -- Attachment: HBASE-17329.master.001.patch > Improve message when CallQueueTooBigException > - > > Key: HBASE-17329 > URL: https://issues.apache.org/jira/browse/HBASE-17329 > Project: HBase > Issue Type: Improvement >Reporter: stack >Assignee: stack >Priority: Trivial > Attachments: HBASE-17329.master.001.patch > > > Print out queue item counts and queue sizes. While here correct queue name > spellings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17329) Improve message when CallQueueTooBigException
[ https://issues.apache.org/jira/browse/HBASE-17329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17329: -- Status: Patch Available (was: Open) > Improve message when CallQueueTooBigException > - > > Key: HBASE-17329 > URL: https://issues.apache.org/jira/browse/HBASE-17329 > Project: HBase > Issue Type: Improvement >Reporter: stack >Assignee: stack >Priority: Trivial > Attachments: HBASE-17329.master.001.patch > > > Print out queue item counts and queue sizes. While here correct queue name > spellings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17323) TestAsyncGetMultiThread fails in master
[
https://issues.apache.org/jira/browse/HBASE-17323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755829#comment-15755829
]
stack commented on HBASE-17323:
---
HBASE-17329 adds some info around exceptions thrown when Qs are full. Probably
shouldn't be up against limit but doesn't seem to be the issue. We seem to
misread randomly (HBASE-17329 added logging name of problem row) so something
more substantial going on here I'd say. What you reckon [~Apache9] ? Thanks sir.
> TestAsyncGetMultiThread fails in master
> ---
>
> Key: HBASE-17323
> URL: https://issues.apache.org/jira/browse/HBASE-17323
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
> Attachments: 17323.hack.txt, testAsyncGetMultiThread-output.gz
>
>
> From
> https://builds.apache.org/job/HBase-Trunk_matrix/2137/jdk=JDK%201.8%20(latest),label=Hadoop/testReport/junit/org.apache.hadoop.hbase.client/TestAsyncGetMultiThread/test/
> :
> {code}
> java.util.concurrent.ExecutionException: java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.Bytes.toInt(Bytes.java:1003)
> at org.apache.hadoop.hbase.util.Bytes.toInt(Bytes.java:980)
> at
> org.apache.hadoop.hbase.client.TestAsyncGetMultiThread.run(TestAsyncGetMultiThread.java:108)
> at
> org.apache.hadoop.hbase.client.TestAsyncGetMultiThread.lambda$null$1(TestAsyncGetMultiThread.java:122)
> {code}
> This can be reproduced locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17282) Reduce the redundant requests to meta table
[ https://issues.apache.org/jira/browse/HBASE-17282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755843#comment-15755843 ] stack commented on HBASE-17282: --- I +1'd the patch over on RB. Its easy to follow what is going on. One comment though, looking over in HBASE-17323, this stuff is noisy. See logs from a run of TestAsyncGetMultiThread. You might want to clean some of it up on commit. > Reduce the redundant requests to meta table > --- > > Key: HBASE-17282 > URL: https://issues.apache.org/jira/browse/HBASE-17282 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17282-v1.patch, HBASE-17282.patch > > > This usually happens at startup when the meta cache is empty. There will be a > lot of locating requests, but most of will have same results. Things become > worse if we do batch operations with AsyncTable as we will send a locating > request for each operation concurrently. > We need to reduce the redundant requests to meta table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15432) TableInputFormat - support multiple column families scan
[ https://issues.apache.org/jira/browse/HBASE-15432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755871#comment-15755871 ] Xuesen Liang commented on HBASE-15432: -- Hi [~te...@apache.org], the unit test is passed. Can you review HBASE-15432.master.003.patch and commit it? Thanks~ > TableInputFormat - support multiple column families scan > > > Key: HBASE-15432 > URL: https://issues.apache.org/jira/browse/HBASE-15432 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0, 0.98.16, 0.98.16.1 >Reporter: nirav patel >Assignee: Xuesen Liang > Fix For: 2.0.0 > > Attachments: HBASE-15432.master.002.patch, > HBASE-15432.master.003.patch, HBASE-15432.master.patch > > > Currently Hbase TableInputFormat class has SCAN_COLUMN_FAMILY and > SCAN_COLUMNS. SCAN_COLUMN_FAMILY can only scan single column family. If we > need to scan multiple column families from a table then we must use > SCAN_COLUMNS where we must provide both columns and column families which is > not a convenient. Can we have a SCAN_COLUMN_FAMILIES which supports scan of > multiple column families. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17275) Assign timeout cause region unassign forever
[
https://issues.apache.org/jira/browse/HBASE-17275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755887#comment-15755887
]
Stephen Yuan Jiang commented on HBASE-17275:
I will look at all AM-related JIRA [~allan163] mentioned above.
> Assign timeout cause region unassign forever
>
>
> Key: HBASE-17275
> URL: https://issues.apache.org/jira/browse/HBASE-17275
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Affects Versions: 1.2.3, 1.1.7
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17275-branch-1.patch
>
>
> This is a real cased happened in my test cluster.
> I have more 8000 regions to assign when I restart a cluster, but I only
> started one regionserver. That means master need to assign these 8000 regions
> to a single server(I know it is not right, but just for testing).
> The rs recevied the open region rpc and began to open regions. But the due to
> the hugh number of regions, , master timeout the rpc call(but actually some
> region had already opened) after 1 mins, as you can see from log 1.
> {noformat}
> 1. 2016-11-22 10:17:32,285 INFO [example.org:30001.activeMasterManager]
> master.AssignmentManager: Unable to communicate with
> example.org,30003,1479780976834 in order to assign regions,
> java.io.IOException: Call to /example.org:30003 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=1, waitTime=60001,
> operationTimeout=6 expired.
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1338)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1272)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:216)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:290)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:30177)
> at
> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:1000)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1719)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2828)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2775)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assignAllUserRegions(AssignmentManager.java:2876)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.processDeadServersAndRegionsInTransition(AssignmentManager.java:646)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:493)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:796)
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:188)
> at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1711)
> at java.lang.Thread.run(Thread.java:756)
> Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=1,
> waitTime=60001, operationTimeout=6 expired.
> at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:81)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1246)
> ... 14 more
> {noformat}
> for the region 7e9aee32eb98a6fc9d503b99fc5f9615(like many others), after
> timeout, master use a pool to re-assign them, as in 2
> {noformat}
> 2. 2016-11-22 10:17:32,303 DEBUG [AM.-pool1-t26] master.AssignmentManager:
> Force region state offline {7e9aee32eb98a6fc9d503b99fc5f9615
> state=PENDING_OPEN, ts=1479780992078, server=example.org,30003,1479780976834}
>
> {noformat}
> But, this region was actually opened on the rs, but (maybe) due to the hugh
> pressure, the OPENED zk event recevied by master , as you can tell from 3,
> "which is more than 15 seconds late"
> {noformat}
> 3. 2016-11-22 10:17:32,304 DEBUG [AM.ZK.Worker-pool2-t3]
> master.AssignmentManager: Handling RS_ZK_REGION_OPENED,
> server=example.org,30003,1479780976834,
> region=7e9aee32eb98a6fc9d503b99fc5f9615, which is more than 15 seconds late,
> current_state={7e9aee32eb98a6fc9d503b99fc5f9615 state=PENDING_OPEN,
> ts=1479780992078, server=example.org,30003,1479780976834}
> {noformat}
> In the meantime, master still try to re-assign this region in another thread.
> Master first close this region in case of multi assign, then change the state
> of this region change from PENDING_OPEN >OFFLINE>PENDING_OPEN. Its RIT node
> in zk was also transitioned to OFFLINE, as in 4,5,6,7
> {noformat}
> 4. 2016-11-22 10:17:32,321 DEBUG [AM.-poo
[jira] [Commented] (HBASE-17149) Procedure v2 - Fix nonce submission
[ https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755900#comment-15755900 ] Stephen Yuan Jiang commented on HBASE-17149: [~stack], since the only UT failure is unrelated to this patch, let us just commit it. Also, branch-1 branch code is a little different from master branch; do you want me to create a branch-1 patch? > Procedure v2 - Fix nonce submission > --- > > Key: HBASE-17149 > URL: https://issues.apache.org/jira/browse/HBASE-17149 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Attachments: HBASE-17149.master.001.patch, > HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, > HBASE-17149.master.002.patch, nonce.patch > > > instead of having all the logic in submitProcedure(), split in > registerNonce() + submitProcedure(). > In this case we can avoid calling the coprocessor twice and having a clean > submit logic knowing that there will only be one submission. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-17220) [C++] Address major issues from cpplint
[
https://issues.apache.org/jira/browse/HBASE-17220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar resolved HBASE-17220.
---
Resolution: Fixed
Assignee: Enis Soztutar
Fix Version/s: HBASE-14850
I've pushed this to the branch to get things going.
> [C++] Address major issues from cpplint
>
>
> Key: HBASE-17220
> URL: https://issues.apache.org/jira/browse/HBASE-17220
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: HBASE-14850
>
> Attachments: hbase-17220_v1.patch
>
>
> See HBASE-17218.
> Some warnings seems important to address:
> {code}
> core/time_range.cc:29: Use int16/int64/etc, rather than the C type long
> [runtime/int] [4]
> core/cell.h:39: Use int16/int64/etc, rather than the C type long
> [runtime/int] [4]
> core/cell.h:45: Use int16/int64/etc, rather than the C type long
> [runtime/int] [4]
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15432) TableInputFormat - support multiple column families scan
[
https://issues.apache.org/jira/browse/HBASE-15432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755912#comment-15755912
]
Ted Yu commented on HBASE-15432:
{code}
+ args = new String[] { "-D " + TableInputFormat.SCAN_COLUMN_FAMILY + "=a,
b",
{code}
Is the space intentional ?
The space doesn't appear in usage.
> TableInputFormat - support multiple column families scan
>
>
> Key: HBASE-15432
> URL: https://issues.apache.org/jira/browse/HBASE-15432
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 2.0.0, 0.98.16, 0.98.16.1
>Reporter: nirav patel
>Assignee: Xuesen Liang
> Fix For: 2.0.0
>
> Attachments: HBASE-15432.master.002.patch,
> HBASE-15432.master.003.patch, HBASE-15432.master.patch
>
>
> Currently Hbase TableInputFormat class has SCAN_COLUMN_FAMILY and
> SCAN_COLUMNS. SCAN_COLUMN_FAMILY can only scan single column family. If we
> need to scan multiple column families from a table then we must use
> SCAN_COLUMNS where we must provide both columns and column families which is
> not a convenient. Can we have a SCAN_COLUMN_FAMILIES which supports scan of
> multiple column families.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17149) Procedure v2 - Fix nonce submission
[ https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755918#comment-15755918 ] stack commented on HBASE-17149: --- Sorry. Detour tryinng to fix the above failing UT. It fails half-the-time. Looking now at this patch to see if bulk insert needs nonce too. > Procedure v2 - Fix nonce submission > --- > > Key: HBASE-17149 > URL: https://issues.apache.org/jira/browse/HBASE-17149 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Attachments: HBASE-17149.master.001.patch, > HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, > HBASE-17149.master.002.patch, nonce.patch > > > instead of having all the logic in submitProcedure(), split in > registerNonce() + submitProcedure(). > In this case we can avoid calling the coprocessor twice and having a clean > submit logic knowing that there will only be one submission. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17328) Properly dispose of looped replication peers
[
https://issues.apache.org/jira/browse/HBASE-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vincent Poon updated HBASE-17328:
-
Attachment: HBASE-17328-master.v1.patch
HBASE-17328-1.1.v1.patch
> Properly dispose of looped replication peers
>
>
> Key: HBASE-17328
> URL: https://issues.apache.org/jira/browse/HBASE-17328
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.4.0, 0.98.23
>Reporter: Vincent Poon
>Assignee: Vincent Poon
> Attachments: HBASE-17328-1.1.v1.patch, HBASE-17328-master.v1.patch
>
>
> When adding a looped replication peer (clusterId == peerClusterId), the
> following code terminates the replication source thread, but since the source
> manager still holds a reference, WALs continue to get enqueued, and never get
> cleaned because they're stuck in the queue, leading to an unsustainable
> buildup. Furthermore, the replication statistics thread will continue to
> print statistics for the terminated source.
> {code}
> if (clusterId.equals(peerClusterId) &&
> !replicationEndpoint.canReplicateToSameCluster()) {
> this.terminate("ClusterId " + clusterId + " is replicating to itself:
> peerClusterId "
> + peerClusterId + " which is not allowed by ReplicationEndpoint:"
> + replicationEndpoint.getClass().getName(), null, false);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17328) Properly dispose of looped replication peers
[
https://issues.apache.org/jira/browse/HBASE-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vincent Poon updated HBASE-17328:
-
Status: Patch Available (was: Open)
> Properly dispose of looped replication peers
>
>
> Key: HBASE-17328
> URL: https://issues.apache.org/jira/browse/HBASE-17328
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 0.98.23, 2.0.0, 1.4.0
>Reporter: Vincent Poon
>Assignee: Vincent Poon
> Attachments: HBASE-17328-1.1.v1.patch, HBASE-17328-master.v1.patch
>
>
> When adding a looped replication peer (clusterId == peerClusterId), the
> following code terminates the replication source thread, but since the source
> manager still holds a reference, WALs continue to get enqueued, and never get
> cleaned because they're stuck in the queue, leading to an unsustainable
> buildup. Furthermore, the replication statistics thread will continue to
> print statistics for the terminated source.
> {code}
> if (clusterId.equals(peerClusterId) &&
> !replicationEndpoint.canReplicateToSameCluster()) {
> this.terminate("ClusterId " + clusterId + " is replicating to itself:
> peerClusterId "
> + peerClusterId + " which is not allowed by ReplicationEndpoint:"
> + replicationEndpoint.getClass().getName(), null, false);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17280) Add mechanism to control hbase cleaner behavior
[
https://issues.apache.org/jira/browse/HBASE-17280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755935#comment-15755935
]
Ted Yu commented on HBASE-17280:
{code}
+return stub.enableCleaner(controller, request);
{code}
Currently we have:
{code}
public boolean setBalancerRunning(final boolean on, final boolean synchronous)
{code}
How about naming the new method setCleanerRunning ?
{code}
+ boolean isCleanerEnabled() {
+if (hfileCleaner != null && logCleaner != null) {
+ return hfileCleaner.getEnabled() && logCleaner.getEnabled();
{code}
Should hfile and log cleaners get their own flag ?
{code}
+ public RunCleanerResponse runCleaner(RpcController c, RunCleanerRequest req)
...
+ Boolean result = master.getHFileCleaner().runCleaner()
+ && master.getLogCleaner().runCleaner();
{code}
No need to check whether the cleaners are enabled ?
{code}
+} else {
+ LOG.warn("Cleaner disabled! Not cleaning.");
{code}
Would the above flood log file ?
> Add mechanism to control hbase cleaner behavior
> ---
>
> Key: HBASE-17280
> URL: https://issues.apache.org/jira/browse/HBASE-17280
> Project: HBase
> Issue Type: Improvement
> Components: Client, hbase, shell
>Affects Versions: 2.0.0, 1.2.0
>Reporter: Ajay Jadhav
>Priority: Minor
> Fix For: 2.0.0, 1.2.0
>
> Attachments: HBASE-17280.branch-1.2.patch,
> HBASE-17280.branch-2.0.patch
>
>
> Cleaner is used to get rid of archived HFiles and old WALs in HBase.
> In the case of heavy workload, cleaner can affect query performance by
> creating a lot of connections to perform costly reads/ writes against
> underlying filesystem.
> This patch allows the user to control HBase cleaner behavior by providing
> shell commands to enable/ disable and manually run it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17328) Properly dispose of looped replication peers
[
https://issues.apache.org/jira/browse/HBASE-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755940#comment-15755940
]
Ted Yu commented on HBASE-17328:
{code}
330 this.manager.removeSource(this);
331 this.replicationQueues.removeQueue(peerId);
332 uninitialize();
333 return;
{code}
The above should be placed ahead of the terminate() call, right ?
> Properly dispose of looped replication peers
>
>
> Key: HBASE-17328
> URL: https://issues.apache.org/jira/browse/HBASE-17328
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.4.0, 0.98.23
>Reporter: Vincent Poon
>Assignee: Vincent Poon
> Attachments: HBASE-17328-1.1.v1.patch, HBASE-17328-master.v1.patch
>
>
> When adding a looped replication peer (clusterId == peerClusterId), the
> following code terminates the replication source thread, but since the source
> manager still holds a reference, WALs continue to get enqueued, and never get
> cleaned because they're stuck in the queue, leading to an unsustainable
> buildup. Furthermore, the replication statistics thread will continue to
> print statistics for the terminated source.
> {code}
> if (clusterId.equals(peerClusterId) &&
> !replicationEndpoint.canReplicateToSameCluster()) {
> this.terminate("ClusterId " + clusterId + " is replicating to itself:
> peerClusterId "
> + peerClusterId + " which is not allowed by ReplicationEndpoint:"
> + replicationEndpoint.getClass().getName(), null, false);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17174) Refactor the AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-17174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-17174: -- Status: Open (was: Patch Available) Nothing happened... retry > Refactor the AsyncProcess > - > > Key: HBASE-17174 > URL: https://issues.apache.org/jira/browse/HBASE-17174 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17174.v0.patch, HBASE-17174.v1.patch, > HBASE-17174.v10.patch, HBASE-17174.v11.patch, HBASE-17174.v2.patch, > HBASE-17174.v3.patch, HBASE-17174.v4.patch, HBASE-17174.v5.patch, > HBASE-17174.v6.patch, HBASE-17174.v7.patch, HBASE-17174.v8.patch, > HBASE-17174.v9.patch > > > The following are reasons of reusing the pool and AP. > # A update-heavy application, for example, loader, creates many > BufferedMutator for batch updates. But these BufferedMutators can’t share a > large threadpool because the shutdown() method will be called when closing > any BufferedMutator. This patch adds a flag into BufferedMutatorParams for > preventing calling the shutdown() method in BufferedMutatorImpl#close > # The AsyncProcess has the powerful traffic control, but the control is > against the single table currently. We should allow alternate traffic control > implementation for advanced user who want more control. > All suggestions are welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17174) Refactor the AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-17174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-17174: -- Description: The following are reasons of refactoring the AP. # A update-heavy application, for example, loader, creates many BufferedMutator for batch updates. But these BufferedMutators can’t share a large threadpool because the shutdown() method will be called when closing any BufferedMutator. This patch adds a flag into BufferedMutatorParams for preventing calling the shutdown() method in BufferedMutatorImpl#close # The AsyncProcess has the powerful traffic control, but the control is against the single table currently. We should allow alternate traffic control implementation for advanced user who want more control. All suggestions are welcome. was: The following are reasons of reusing the pool and AP. # A update-heavy application, for example, loader, creates many BufferedMutator for batch updates. But these BufferedMutators can’t share a large threadpool because the shutdown() method will be called when closing any BufferedMutator. This patch adds a flag into BufferedMutatorParams for preventing calling the shutdown() method in BufferedMutatorImpl#close # The AsyncProcess has the powerful traffic control, but the control is against the single table currently. We should allow alternate traffic control implementation for advanced user who want more control. All suggestions are welcome. > Refactor the AsyncProcess > - > > Key: HBASE-17174 > URL: https://issues.apache.org/jira/browse/HBASE-17174 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17174.v0.patch, HBASE-17174.v1.patch, > HBASE-17174.v10.patch, HBASE-17174.v11.patch, HBASE-17174.v2.patch, > HBASE-17174.v3.patch, HBASE-17174.v4.patch, HBASE-17174.v5.patch, > HBASE-17174.v6.patch, HBASE-17174.v7.patch, HBASE-17174.v8.patch, > HBASE-17174.v9.patch > > > The following are reasons of refactoring the AP. > # A update-heavy application, for example, loader, creates many > BufferedMutator for batch updates. But these BufferedMutators can’t share a > large threadpool because the shutdown() method will be called when closing > any BufferedMutator. This patch adds a flag into BufferedMutatorParams for > preventing calling the shutdown() method in BufferedMutatorImpl#close > # The AsyncProcess has the powerful traffic control, but the control is > against the single table currently. We should allow alternate traffic control > implementation for advanced user who want more control. > All suggestions are welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17323) TestAsyncGetMultiThread fails in master
[
https://issues.apache.org/jira/browse/HBASE-17323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755944#comment-15755944
]
Duo Zhang commented on HBASE-17323:
---
I think the queue full is in the setUp method where I put 1000 rows
concurrently. This is what [~carp84] said, with async client we need to pay
more attention on the pressure of RS. One simple solution is to use a separated
sleep mechanism when call queue is full, and a more beautiful solution is to
track the pressure of RS and give a reasonable sleep time when retrying. We
have done some works before, see the classes under o.a.h.h.client.backoff
package, but I haven't found a proper way to use them. But I do not think this
is the reason why this test fails. It just wastes some time...
And the error posted by Ted is intenional. In this test I keep moving
regions(include meta region) across RSes to test if there are data races which
will lead to incorrect result. Let me dig more.
Thanks.
> TestAsyncGetMultiThread fails in master
> ---
>
> Key: HBASE-17323
> URL: https://issues.apache.org/jira/browse/HBASE-17323
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
> Attachments: 17323.hack.txt, testAsyncGetMultiThread-output.gz
>
>
> From
> https://builds.apache.org/job/HBase-Trunk_matrix/2137/jdk=JDK%201.8%20(latest),label=Hadoop/testReport/junit/org.apache.hadoop.hbase.client/TestAsyncGetMultiThread/test/
> :
> {code}
> java.util.concurrent.ExecutionException: java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.Bytes.toInt(Bytes.java:1003)
> at org.apache.hadoop.hbase.util.Bytes.toInt(Bytes.java:980)
> at
> org.apache.hadoop.hbase.client.TestAsyncGetMultiThread.run(TestAsyncGetMultiThread.java:108)
> at
> org.apache.hadoop.hbase.client.TestAsyncGetMultiThread.lambda$null$1(TestAsyncGetMultiThread.java:122)
> {code}
> This can be reproduced locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17174) Refactor the AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-17174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-17174: -- Attachment: HBASE-17174.v11.patch > Refactor the AsyncProcess > - > > Key: HBASE-17174 > URL: https://issues.apache.org/jira/browse/HBASE-17174 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17174.v0.patch, HBASE-17174.v1.patch, > HBASE-17174.v10.patch, HBASE-17174.v11.patch, HBASE-17174.v11.patch, > HBASE-17174.v2.patch, HBASE-17174.v3.patch, HBASE-17174.v4.patch, > HBASE-17174.v5.patch, HBASE-17174.v6.patch, HBASE-17174.v7.patch, > HBASE-17174.v8.patch, HBASE-17174.v9.patch > > > The following are reasons of refactoring the AP. > # A update-heavy application, for example, loader, creates many > BufferedMutator for batch updates. But these BufferedMutators can’t share a > large threadpool because the shutdown() method will be called when closing > any BufferedMutator. This patch adds a flag into BufferedMutatorParams for > preventing calling the shutdown() method in BufferedMutatorImpl#close > # The AsyncProcess has the powerful traffic control, but the control is > against the single table currently. We should allow alternate traffic control > implementation for advanced user who want more control. > All suggestions are welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17174) Refactor the AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-17174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-17174: -- Status: Patch Available (was: Open) > Refactor the AsyncProcess > - > > Key: HBASE-17174 > URL: https://issues.apache.org/jira/browse/HBASE-17174 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17174.v0.patch, HBASE-17174.v1.patch, > HBASE-17174.v10.patch, HBASE-17174.v11.patch, HBASE-17174.v11.patch, > HBASE-17174.v2.patch, HBASE-17174.v3.patch, HBASE-17174.v4.patch, > HBASE-17174.v5.patch, HBASE-17174.v6.patch, HBASE-17174.v7.patch, > HBASE-17174.v8.patch, HBASE-17174.v9.patch > > > The following are reasons of refactoring the AP. > # A update-heavy application, for example, loader, creates many > BufferedMutator for batch updates. But these BufferedMutators can’t share a > large threadpool because the shutdown() method will be called when closing > any BufferedMutator. This patch adds a flag into BufferedMutatorParams for > preventing calling the shutdown() method in BufferedMutatorImpl#close > # The AsyncProcess has the powerful traffic control, but the control is > against the single table currently. We should allow alternate traffic control > implementation for advanced user who want more control. > All suggestions are welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17329) Improve message when CallQueueTooBigException
[
https://issues.apache.org/jira/browse/HBASE-17329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755951#comment-15755951
]
Duo Zhang commented on HBASE-17329:
---
Yeah this is better for debug, but I think the original approach of using the
same {{CALL_QUEUE_TOO_BIG_EXCEPTION}} is to save the time of generating
stacktrace. Maybe the author thinks performance is a big problem here?
> Improve message when CallQueueTooBigException
> -
>
> Key: HBASE-17329
> URL: https://issues.apache.org/jira/browse/HBASE-17329
> Project: HBase
> Issue Type: Improvement
>Reporter: stack
>Assignee: stack
>Priority: Trivial
> Attachments: HBASE-17329.master.001.patch
>
>
> Print out queue item counts and queue sizes. While here correct queue name
> spellings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17323) TestAsyncGetMultiThread fails in master
[
https://issues.apache.org/jira/browse/HBASE-17323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755956#comment-15755956
]
stack commented on HBASE-17323:
---
Yeah, on CallQueueTooBigException not being the issue because I got rid of
exception by making Q massive and it still failed. Yeah, on pushback.
The NPE happens randomly. Could we be mis-reading? Dropping value. Seems to
happen at random places doing Get of 1k. [~Apache9]
> TestAsyncGetMultiThread fails in master
> ---
>
> Key: HBASE-17323
> URL: https://issues.apache.org/jira/browse/HBASE-17323
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
> Attachments: 17323.hack.txt, testAsyncGetMultiThread-output.gz
>
>
> From
> https://builds.apache.org/job/HBase-Trunk_matrix/2137/jdk=JDK%201.8%20(latest),label=Hadoop/testReport/junit/org.apache.hadoop.hbase.client/TestAsyncGetMultiThread/test/
> :
> {code}
> java.util.concurrent.ExecutionException: java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.Bytes.toInt(Bytes.java:1003)
> at org.apache.hadoop.hbase.util.Bytes.toInt(Bytes.java:980)
> at
> org.apache.hadoop.hbase.client.TestAsyncGetMultiThread.run(TestAsyncGetMultiThread.java:108)
> at
> org.apache.hadoop.hbase.client.TestAsyncGetMultiThread.lambda$null$1(TestAsyncGetMultiThread.java:122)
> {code}
> This can be reproduced locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17149) Procedure v2 - Fix nonce submission
[ https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17149: -- Attachment: HBASE-17149.master.003.patch > Procedure v2 - Fix nonce submission > --- > > Key: HBASE-17149 > URL: https://issues.apache.org/jira/browse/HBASE-17149 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Attachments: HBASE-17149.master.001.patch, > HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, > HBASE-17149.master.002.patch, HBASE-17149.master.003.patch, nonce.patch > > > instead of having all the logic in submitProcedure(), split in > registerNonce() + submitProcedure(). > In this case we can avoid calling the coprocessor twice and having a clean > submit logic knowing that there will only be one submission. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17149) Procedure v2 - Fix nonce submission
[ https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755960#comment-15755960 ] stack commented on HBASE-17149: --- Rebase. Does the bulk submitProcedure need to handle NONCE? I added note... > Procedure v2 - Fix nonce submission > --- > > Key: HBASE-17149 > URL: https://issues.apache.org/jira/browse/HBASE-17149 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Attachments: HBASE-17149.master.001.patch, > HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, > HBASE-17149.master.002.patch, HBASE-17149.master.003.patch, nonce.patch > > > instead of having all the logic in submitProcedure(), split in > registerNonce() + submitProcedure(). > In this case we can avoid calling the coprocessor twice and having a clean > submit logic knowing that there will only be one submission. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17149) Procedure v2 - Fix nonce submission
[ https://issues.apache.org/jira/browse/HBASE-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755962#comment-15755962 ] stack commented on HBASE-17149: --- Currently we don't do bulk submits so it is for later. Lets keep an eye on it. > Procedure v2 - Fix nonce submission > --- > > Key: HBASE-17149 > URL: https://issues.apache.org/jira/browse/HBASE-17149 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Attachments: HBASE-17149.master.001.patch, > HBASE-17149.master.002.patch, HBASE-17149.master.002.patch, > HBASE-17149.master.002.patch, HBASE-17149.master.003.patch, nonce.patch > > > instead of having all the logic in submitProcedure(), split in > registerNonce() + submitProcedure(). > In this case we can avoid calling the coprocessor twice and having a clean > submit logic knowing that there will only be one submission. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17329) Improve message when CallQueueTooBigException
[
https://issues.apache.org/jira/browse/HBASE-17329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755974#comment-15755974
]
stack commented on HBASE-17329:
---
Was added by:
{code}
Author: Elliott Clark
Date: Wed Apr 15 13:49:55 2015 -0700
HBASE-13477 Create metrics on failed requests
Summary: Add metrics on how many requests are exceptions and what type.
Test Plan: behold unit tests.
Differential Revision: https://reviews.facebook.net/D37167
{code}
Nothing on why. I'd imagine that when we are throwing these if lots of clients,
yeah, we could be making loads of them.
Its nice being able to see what queue is full and by how much.
Let me leave this patch here until gets more petitioners.
> Improve message when CallQueueTooBigException
> -
>
> Key: HBASE-17329
> URL: https://issues.apache.org/jira/browse/HBASE-17329
> Project: HBase
> Issue Type: Improvement
>Reporter: stack
>Assignee: stack
>Priority: Trivial
> Attachments: HBASE-17329.master.001.patch
>
>
> Print out queue item counts and queue sizes. While here correct queue name
> spellings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15432) TableInputFormat - support multiple column families scan
[ https://issues.apache.org/jira/browse/HBASE-15432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755980#comment-15755980 ] Xuesen Liang commented on HBASE-15432: -- Yes, the space is intentional. It's for testing that if user write whitespace, it still work fine. Both "=a, b" and "=a,b" are fine. > TableInputFormat - support multiple column families scan > > > Key: HBASE-15432 > URL: https://issues.apache.org/jira/browse/HBASE-15432 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0, 0.98.16, 0.98.16.1 >Reporter: nirav patel >Assignee: Xuesen Liang > Fix For: 2.0.0 > > Attachments: HBASE-15432.master.002.patch, > HBASE-15432.master.003.patch, HBASE-15432.master.patch > > > Currently Hbase TableInputFormat class has SCAN_COLUMN_FAMILY and > SCAN_COLUMNS. SCAN_COLUMN_FAMILY can only scan single column family. If we > need to scan multiple column families from a table then we must use > SCAN_COLUMNS where we must provide both columns and column families which is > not a convenient. Can we have a SCAN_COLUMN_FAMILIES which supports scan of > multiple column families. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17257) Add column-aliasing capability to hbase-client
[ https://issues.apache.org/jira/browse/HBASE-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755984#comment-15755984 ] Daniel Vimont commented on HBASE-17257: --- One of the errors above is due to a new test module (apparently introduced in HBASE-17277), TestBufferedMutator in hbase-client, which uses a dummied-out Connection (with its own "DoNothingRegistry" class), apparently in order to facilitate unit testing on ConnectionImplementation#getBufferedMutator without the need for usage of the HBaseTestingUtility. Would it be more appropriate for the TestBufferedMutator test module to be placed over in hbase-server's Test directory with all the other hbase-client test modules which need to utilize HBaseTestingUtility (and remove the "dummy" logic from TestBufferedMutator)? This would prevent not only the error that I'm encountering here, but other errors that might be encountered in the future as #getBufferedMutator might need to be extended. > Add column-aliasing capability to hbase-client > -- > > Key: HBASE-17257 > URL: https://issues.apache.org/jira/browse/HBASE-17257 > Project: HBase > Issue Type: New Feature > Components: Client >Affects Versions: 2.0.0 >Reporter: Daniel Vimont >Assignee: Daniel Vimont > Labels: features > Attachments: HBASE-17257-v2.patch, HBASE-17257-v3.patch, > HBASE-17257-v4.patch, HBASE-17257.patch > > > Review Board link: https://reviews.apache.org/r/54635/ > Column aliasing will provide the option for a 1, 2, or 4 byte alias value to > be stored in each cell of an "alias enabled" column-family, in place of the > full-length column-qualifier. Aliasing is intended to operate completely > invisibly to the end-user developer, with absolutely no "awareness" of > aliasing required to be coded into a front-end application. No new public > hbase-client interfaces are to be introduced, and only a few new public > methods should need to be added to existing interfaces, primarily to allow an > administrator to designate that a new column-family is to be alias-enabled by > setting its aliasSize attribute to 1, 2, or 4. > To facilitate such functionality, new subclasses of HTable, > BufferedMutatorImpl, and HTableMultiplexer are to be provided. The overriding > methods of these new subclasses will invoke methods of the new AliasManager > class to facilitate qualifier-to-alias conversions (for user-submitted Gets, > Scans, and Mutations) and alias-to-qualifier conversions (for Results > returned from HBase) for any Table that has one or more alias-enabled column > families. All conversion logic will be encapsulated in the new AliasManager > class, and all qualifier-to-alias mappings will be persisted in a new > aliasMappingTable in a new, reserved namespace. > An informal polling of HBase users at HBaseCon East and at the > Strata/Hadoop-World conference in Sept. 2016 showed that Column Aliasing > could be a popular enhancement to standard HBase functionality, due to the > fact that full column-qualifiers are stored in each cell, and reducing this > qualifier storage requirement down to 1, 2, or 4 bytes per cell could prove > beneficial in terms of reduced storage and bandwidth needs. Aliasing is > intended chiefly for column-families which are of the "narrow and tall" > variety (i.e., that are designed to use relatively few distinct > column-qualifiers throughout a large number of rows, throughout the lifespan > of the column-family). A column-family that is set up with an alias-size of 1 > byte can contain up to 255 unique column-qualifiers; a 2 byte alias-size > allows for up to 65,535 unique column-qualifiers; and a 4 byte alias-size > allows for up to 4,294,967,295 unique column-qualifiers. > Fuller specifications will be entered into the comments section below. Note > that it may well not be viable to add aliasing support in the new "async" > classes that appear to be currently under development. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15432) TableInputFormat - support multiple column families scan
[ https://issues.apache.org/jira/browse/HBASE-15432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755988#comment-15755988 ] Ted Yu commented on HBASE-15432: You can pass "=a, b" using String[] However, using command line, can you do the same ? If so, mind showing me log from a live action ? Thanks > TableInputFormat - support multiple column families scan > > > Key: HBASE-15432 > URL: https://issues.apache.org/jira/browse/HBASE-15432 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0, 0.98.16, 0.98.16.1 >Reporter: nirav patel >Assignee: Xuesen Liang > Fix For: 2.0.0 > > Attachments: HBASE-15432.master.002.patch, > HBASE-15432.master.003.patch, HBASE-15432.master.patch > > > Currently Hbase TableInputFormat class has SCAN_COLUMN_FAMILY and > SCAN_COLUMNS. SCAN_COLUMN_FAMILY can only scan single column family. If we > need to scan multiple column families from a table then we must use > SCAN_COLUMNS where we must provide both columns and column families which is > not a convenient. Can we have a SCAN_COLUMN_FAMILIES which supports scan of > multiple column families. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17327) Allow for lazy connection / BufferedMutator creation
[ https://issues.apache.org/jira/browse/HBASE-17327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755991#comment-15755991 ] stack commented on HBASE-17327: --- Is this because we 'connect' in the Connection constructor? > Allow for lazy connection / BufferedMutator creation > > > Key: HBASE-17327 > URL: https://issues.apache.org/jira/browse/HBASE-17327 > Project: HBase > Issue Type: Sub-task >Reporter: Joep Rottinghuis > > The SpoolingBufferedMutatorImpl solution (HBASE-17018) solved the use-case > when an existing HBase connection stalls out due to HBase being down, or > other transient issues. > We have a remaining issue that our service will not start up if it cannot > initially connect to HBase. > This can be solved by letting the SpoolingBufferedMutator create the wrapped > BufferedMutatorImpl later, but the problem has already occurred: we already > have to have a connection in order to create a BufferedMutator to begin with. > It would be good to be able to initiate a connection and then have a way for > multiple users to wait for the connection to succeed before using it. > I'm thinking perhaps create a LazyConnection interface that extends the > Connection interface. It would have an additional waitFor(long timeout, > TimeUnit unit) method where clients can wait for the connection to be > established before they start using the connection. > Or perhaps the ConnectionFactory can have a createLazyConnection method. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16859) Use Bytebuffer pool for non java clients specifically for scans/gets
[ https://issues.apache.org/jira/browse/HBASE-16859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756000#comment-15756000 ] stack commented on HBASE-16859: --- Are the failures related at all? Otherwise, patch LGTM. Lets try it. > Use Bytebuffer pool for non java clients specifically for scans/gets > > > Key: HBASE-16859 > URL: https://issues.apache.org/jira/browse/HBASE-16859 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-16859_V1.patch, HBASE-16859_V2.patch, > HBASE-16859_V2.patch, HBASE-16859_V4.patch, HBASE-16859_V5.patch > > > In case of non java clients we still write the results and header into a on > demand byte[]. This can be changed to use the BBPool (onheap or offheap > buffer?). > But the basic problem is to identify if the response is for scans/gets. > - One easy way to do it is use the MethodDescriptor per Call and use the > name of the MethodDescriptor to identify it is a scan/get. But this will > pollute RpcServer by checking for scan/get type response. > - Other way is always set the result to cellScanner but we know that > isClientCellBlockSupported is going to false for non PB clients. So ignore > the cellscanner and go ahead with the results in PB. But this is not clean > - third one is that we already have a RpccallContext being passed to the RS. > In case of scan/gets/multiGets we already set a Rpccallback for shipped call. > So here on response we can check if the callback is not null and check for > isclientBlockSupported. In this case we can get the BB from the pool and > write the result and header to that BB. May be this looks clean? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17081) Flush the entire CompactingMemStore content to disk
[ https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15756019#comment-15756019 ] stack commented on HBASE-17081: --- You can see it in here that it fails pretty frequently: https://builds.apache.org/job/HBASE-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html > Flush the entire CompactingMemStore content to disk > --- > > Key: HBASE-17081 > URL: https://issues.apache.org/jira/browse/HBASE-17081 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Attachments: HBASE-15787_8.patch, HBASE-17081-V01.patch, > HBASE-17081-V02.patch, HBASE-17081-V03.patch, HBASE-17081-V04.patch, > HBASE-17081-V05.patch, HBASE-17081-V06.patch, HBASE-17081-V06.patch, > HBASE-17081-V07.patch, HBaseMeetupDecember2016-V02.pptx, > Pipelinememstore_fortrunk_3.patch > > > Part of CompactingMemStore's memory is held by an active segment, and another > part is divided between immutable segments in the compacting pipeline. Upon > flush-to-disk request we want to flush all of it to disk, in contrast to > flushing only tail of the compacting pipeline. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
