[jira] [Commented] (HBASE-17100) Implement Chore to sync FN info from Master to RegionServers
[ https://issues.apache.org/jira/browse/HBASE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15734624#comment-15734624 ] Devaraj Das commented on HBASE-17100: - [~thiruvel] which RPC failure are you worried about? I just checked the design doc as well, and didn't see a reference to this chore. > Implement Chore to sync FN info from Master to RegionServers > > > Key: HBASE-17100 > URL: https://issues.apache.org/jira/browse/HBASE-17100 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-17100.master.001.patch, HBASE_17100_draft.patch > > > Master will have a repair chore which will periodically sync fn information > from master to all the region servers. This will protect against rpc failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-1935) Scan in parallel
[ https://issues.apache.org/jira/browse/HBASE-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15734614#comment-15734614 ] Xiang Li commented on HBASE-1935: - Hi [~stack], I am quite interested on this, and would like to follow the work based on yours and Lars(in HBASE-9272) to see if I can do something. I am reading code and comment history of both JIRAs, and there might be a long list of questions to bother you, thanks you in advance! I am also reading HBASE-16833 (about async client by Duo Zhang) as I feel I can learn some ideas of it. 1 quick question: Could you please share the reason why you stopped at that time? Please feel free to comment if you have any! > Scan in parallel > > > Key: HBASE-1935 > URL: https://issues.apache.org/jira/browse/HBASE-1935 > Project: HBase > Issue Type: New Feature > Components: Coprocessors >Reporter: stack > Attachments: 1935-idea.txt, pscanner-v2.patch, pscanner-v3.patch, > pscanner-v4.patch, pscanner.patch > > > A scanner that rather than scan in series, instead scanned multiple regions > in parallell would be more involved but could complete much faster > partiularly if results are sparse. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16888) Avoid unnecessary tags copy in Append
[
https://issues.apache.org/jira/browse/HBASE-16888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15734543#comment-15734543
]
ChiaPing Tsai commented on HBASE-16888:
---
TestBlockEvictionFromClient pass locally.
> Avoid unnecessary tags copy in Append
> -
>
> Key: HBASE-16888
> URL: https://issues.apache.org/jira/browse/HBASE-16888
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16888.v0.patch, HBASE-16888.v1.patch,
> HBASE-16888.v1.patch
>
>
> a) If the delta has tags and the mutation doesn’t apply the TTL, we shouldn’t
> create the TagRewriteCell.
> {noformat}
> List tags = TagUtil.carryForwardTags(delta);
> long ts = now;
> Cell newCell = null;
> byte [] row = mutation.getRow();
> if (currentValue != null) {
> ...
> } else {
> // Append's KeyValue.Type==Put and ts==HConstants.LATEST_TIMESTAMP
> CellUtil.updateLatestStamp(delta, now);
> newCell = delta;
> tags = TagUtil.carryForwardTTLTag(tags, mutation.getTTL());
> if (tags != null) {
> newCell = CellUtil.createCell(delta, tags);
> }
> }
> {noformat}
> b) If the cell has tags, the ShareableMemoryTagRewriteCell will make
> duplicate copy of tags.
> {noformat}
> Cell clonedBaseCell = ((ShareableMemory) this.cell).cloneToCell();
> return new TagRewriteCell(clonedBaseCell, this.tags);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17276) Reduce log spam from WrongRegionException in large multi()'s
[
https://issues.apache.org/jira/browse/HBASE-17276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15734497#comment-15734497
]
Hadoop QA commented on HBASE-17276:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 7s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 28s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 10s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 141m 56s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegion |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12842485/HBASE-17276.002.patch
|
| JIRA Issue | HBASE-17276 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux ae964594982f 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / b3ae87b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4854/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4854/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4854/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4854/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Reduce log spam from WrongRegionException in large multi()'s
>
>
>
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15734494#comment-15734494 ] Devaraj Das commented on HBASE-14123: - [~stack], I just did a skim of the RB - your comments and [~vrodionov]'s responses. Seemed okay to me. You had also commented on this jira with a bunch of comments (above), and it seems [~vrodionov] opened follow up jiras for those, and he responded to your comments with the jira numbers inline. I didn't cover each and every comment but overall seemed okay to me. So in summary, what changed in the megapatch is responded to inline in the RB and in the comments above. Does this work for you [~stack]? > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Fix For: 2.0.0 > > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, > 14123-master.v21.txt, 14123-master.v24.txt, 14123-master.v25.txt, > 14123-master.v27.txt, 14123-master.v28.txt, 14123-master.v29.full.txt, > 14123-master.v3.txt, 14123-master.v30.txt, 14123-master.v31.txt, > 14123-master.v32.txt, 14123-master.v33.txt, 14123-master.v34.txt, > 14123-master.v35.txt, 14123-master.v36.txt, 14123-master.v37.txt, > 14123-master.v38.txt, 14123-master.v5.txt, 14123-master.v6.txt, > 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, > 14123.master.v39.patch, 14123.master.v40.patch, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17198) FN updates during region merge (follow up to Procedure v2 merge)
[
https://issues.apache.org/jira/browse/HBASE-17198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15734433#comment-15734433
]
Hadoop QA commented on HBASE-17198:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m 31s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 52s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 88m 47s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 138m 42s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12842481/HBASE-17198.master.001.patch
|
| JIRA Issue | HBASE-17198 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 0703ece5cf9c 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / b3ae87b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4853/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4853/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> FN updates during region merge (follow up to Procedure v2 merge)
>
>
> Key: HBASE-17198
> URL: https://issues.apache.org/jira/browse/HBASE-17198
> Project: HBase
> Issue Type: Sub-task
>Reporter: Thiruvel T
[jira] [Commented] (HBASE-17107) FN info should be cleaned up on region/table cleanup
[
https://issues.apache.org/jira/browse/HBASE-17107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15734378#comment-15734378
]
Hadoop QA commented on HBASE-17107:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 39s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 99m 58s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 146m 13s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12842479/HBASE-17107.master.001.patch
|
| JIRA Issue | HBASE-17107 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 911ffd870d3c 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / b3ae87b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4852/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4852/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> FN info should be cleaned up on region/table cleanup
>
>
> Key: HBASE-17107
> URL: https://issues.apache.org/jira/browse/HBASE-17107
> Project: HBase
> Issue Type: Sub-task
>Reporter: Thiruvel Thirumoolan
>Assignee: Thiruvel Thirumoolan
> Fix For: 2.0.0
>
> Attachments: HBASE-17107.master.001.patch, HBASE_17107.d
[jira] [Commented] (HBASE-16398) optimize HRegion computeHDFSBlocksDistribution
[
https://issues.apache.org/jira/browse/HBASE-16398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15734364#comment-15734364
]
Hadoop QA commented on HBASE-16398:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 40s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 100m 7s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 140m 27s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.mapreduce.TestMultiTableSnapshotInputFormat
|
| | hadoop.hbase.mapred.TestMultiTableSnapshotInputFormat |
| | hadoop.hbase.regionserver.TestHRegion |
| | hadoop.hbase.mapred.TestTableSnapshotInputFormat |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12842480/HBASE-16398_v3.patch |
| JIRA Issue | HBASE-16398 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux e3d8c652fe20 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / b3ae87b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4851/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4851/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4851/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https:
[jira] [Commented] (HBASE-17237) Override the correct compact method in HMobStore
[ https://issues.apache.org/jira/browse/HBASE-17237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15734288#comment-15734288 ] Jingcheng Du commented on HBASE-17237: -- Thanks [~anoop.hbase]! This finding comes from HBASE-16812. The initial purpose by Appy was to remove the unused zk locks. Then I found the compact method was not used properly. Now I split that JIRA into two, implement this urgent one first, and leave the other later. Right, I will pay attention on this. Thanks! > Override the correct compact method in HMobStore > > > Key: HBASE-17237 > URL: https://issues.apache.org/jira/browse/HBASE-17237 > Project: HBase > Issue Type: Bug > Components: mob >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Attachments: HBASE-17237.patch > > > The method Store#compact(CompactionContext compaction, ThroughputController > throughputController) is deprecated now, and HMobStore overrides it. > HMobStore needs to override compact(CompactionContext compaction, > ThroughputController throughputController, User user) instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17276) Reduce log spam from WrongRegionException in large multi()'s
[
https://issues.apache.org/jira/browse/HBASE-17276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-17276:
---
Attachment: HBASE-17276.002.patch
.002 Fixes that whitespace issues that [~busbey]'s astute eyes caught. Thanks!
> Reduce log spam from WrongRegionException in large multi()'s
>
>
> Key: HBASE-17276
> URL: https://issues.apache.org/jira/browse/HBASE-17276
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17276.001.patch, HBASE-17276.002.patch
>
>
> The following spam drives me up a wall in the regionserver log:
> {noformat}
> 2016-12-05 05:53:05,085 WARN
> [RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=16020]
> regionserver.HRegion: Batch mutation had a row that does not belong to this
> region
> org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out
> of range for doMiniBatchMutation on HRegion
> IntegrationTestReplicationSinkRestart,L\xCC\xCC\xCC\xCC\xCC\xCC\xC8,1480916713541.caab3310166699287b54b72b35b29431.,
> startKey='L\xCC\xCC\xCC\xCC\xCC\xCC\xC8',
> getEndKey()='Y\x99\x99\x99\x99\x99\x99\x94',
> row='\x0C\xD2\xA5\xA3\x99\xC7\xE0Q!\x15^\xA6\x90\x1E\xA3\xAD'
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:5211)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkAndPrepareMutation(HRegion.java:3879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3040)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2933)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2875)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:717)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:679)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2056)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32303)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2141)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
> 2016-12-05 05:53:05,086 WARN
> [RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=16020]
> regionserver.HRegion: Batch mutation had a row that does not belong to this
> region
> org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out
> of range for doMiniBatchMutation on HRegion
> IntegrationTestReplicationSinkRestart,L\xCC\xCC\xCC\xCC\xCC\xCC\xC8,1480916713541.caab3310166699287b54b72b35b29431.,
> startKey='L\xCC\xCC\xCC\xCC\xCC\xCC\xC8',
> getEndKey()='Y\x99\x99\x99\x99\x99\x99\x94',
> row='\x0E\xE7\xFA[\x8D\x93;\xF4\xC7F\xF9\x85\x84\x85\xF3\x0E'
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:5211)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkAndPrepareMutation(HRegion.java:3879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3040)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2933)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2875)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:717)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:679)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2056)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32303)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2141)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
> 2016-12-05 05:53:05,087 WARN
> [RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=16020]
> regionserver.HRegion: Batch mutation had a row that does not belong to this
> region
> org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out
> of range for doMiniBatchMutation on HRegion
> IntegrationTestReplicationSinkRestart,L\xCC\xCC\xCC\xCC\xCC\xCC\xC8,1480916713541.caab3310166699287b54b72b35b29431.,
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[
https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15734191#comment-15734191
]
Vladimir Rodionov commented on HBASE-14123:
---
{quote}
Assuming Stack made all the comments on RB, Vladimir can you just respond to
those there, so all the responses are in one place?
{quote}
Hm-m, I have responded already here, to every comment:
https://reviews.apache.org/r/52748/
> HBase Backup/Restore Phase 2
>
>
> Key: HBASE-14123
> URL: https://issues.apache.org/jira/browse/HBASE-14123
> Project: HBase
> Issue Type: Umbrella
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: 14123-master.v14.txt, 14123-master.v15.txt,
> 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt,
> 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt,
> 14123-master.v21.txt, 14123-master.v24.txt, 14123-master.v25.txt,
> 14123-master.v27.txt, 14123-master.v28.txt, 14123-master.v29.full.txt,
> 14123-master.v3.txt, 14123-master.v30.txt, 14123-master.v31.txt,
> 14123-master.v32.txt, 14123-master.v33.txt, 14123-master.v34.txt,
> 14123-master.v35.txt, 14123-master.v36.txt, 14123-master.v37.txt,
> 14123-master.v38.txt, 14123-master.v5.txt, 14123-master.v6.txt,
> 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt,
> 14123.master.v39.patch, 14123.master.v40.patch,
> HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch,
> HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch,
> HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch,
> HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch,
> HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch,
> HBASE-14123-v7.patch, HBASE-14123-v9.patch
>
>
> Phase 2 umbrella JIRA. See HBASE-7912 for design document and description.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16888) Avoid unnecessary tags copy in Append
[
https://issues.apache.org/jira/browse/HBASE-16888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15734170#comment-15734170
]
Hadoop QA commented on HBASE-16888:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
23m 41s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 50s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 32s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 130m 19s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.client.TestBlockEvictionFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12842464/HBASE-16888.v1.patch |
| JIRA Issue | HBASE-16888 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 6efa09d72f5b 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / b3ae87b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4850/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4850/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results
[jira] [Created] (HBASE-17281) FN should use datanode port from hdfs configuration
Thiruvel Thirumoolan created HBASE-17281: Summary: FN should use datanode port from hdfs configuration Key: HBASE-17281 URL: https://issues.apache.org/jira/browse/HBASE-17281 Project: HBase Issue Type: Sub-task Reporter: Thiruvel Thirumoolan Assignee: Thiruvel Thirumoolan Priority: Minor Currently we use the ServerName port for providing favored node hints. We should use the DN port from hdfs-site.xml instead to avoid warning messages in region server logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17198) FN updates during region merge (follow up to Procedure v2 merge)
[ https://issues.apache.org/jira/browse/HBASE-17198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvel Thirumoolan updated HBASE-17198: - Status: Patch Available (was: Open) > FN updates during region merge (follow up to Procedure v2 merge) > > > Key: HBASE-17198 > URL: https://issues.apache.org/jira/browse/HBASE-17198 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-17198.master.001.patch > > > As mentioned in https://reviews.apache.org/r/53242/ (HBASE-16941), since the > procedure v2 merge changes are in development, there is a follow up > optimization/cleanup that can be done for favored nodes during merge. This > jira will be taken up once HBASE-16119 is complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17198) FN updates during region merge (follow up to Procedure v2 merge)
[ https://issues.apache.org/jira/browse/HBASE-17198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvel Thirumoolan updated HBASE-17198: - Attachment: HBASE-17198.master.001.patch > FN updates during region merge (follow up to Procedure v2 merge) > > > Key: HBASE-17198 > URL: https://issues.apache.org/jira/browse/HBASE-17198 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-17198.master.001.patch > > > As mentioned in https://reviews.apache.org/r/53242/ (HBASE-16941), since the > procedure v2 merge changes are in development, there is a follow up > optimization/cleanup that can be done for favored nodes during merge. This > jira will be taken up once HBASE-16119 is complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-9774) Provide a way for coprocessors to register and report custom metrics
[
https://issues.apache.org/jira/browse/HBASE-9774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15734145#comment-15734145
]
Hadoop QA commented on HBASE-9774:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 29s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
44s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hbase-assembly . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 5s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 20s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
21s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 12s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 47s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: .
hbase-assembly {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 32s
{color} | {color:red} hbase-examples generated 2 new + 0 unchanged - 0 fixed =
2 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 8s
{color} | {color:red} hbase-metrics-api generated 4 new + 0 unchanged - 0 fixed
= 4 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 59s
{color} | {color:red} root generated 4 new + 19 unchanged - 0 fixed = 23 total
(was 19) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 39s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 55s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 8s
{color} | {color:green} hbase-metrics-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 16s
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 5m 30s {color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 11s
{color} | {color:green} hbase-assembly in
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15734140#comment-15734140 ] Devaraj Das commented on HBASE-14123: - Assuming Stack made all the comments on RB, Vladimir can you just respond to those there, so all the responses are in one place? > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Fix For: 2.0.0 > > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, > 14123-master.v21.txt, 14123-master.v24.txt, 14123-master.v25.txt, > 14123-master.v27.txt, 14123-master.v28.txt, 14123-master.v29.full.txt, > 14123-master.v3.txt, 14123-master.v30.txt, 14123-master.v31.txt, > 14123-master.v32.txt, 14123-master.v33.txt, 14123-master.v34.txt, > 14123-master.v35.txt, 14123-master.v36.txt, 14123-master.v37.txt, > 14123-master.v38.txt, 14123-master.v5.txt, 14123-master.v6.txt, > 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, > 14123.master.v39.patch, 14123.master.v40.patch, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16398) optimize HRegion computeHDFSBlocksDistribution
[ https://issues.apache.org/jira/browse/HBASE-16398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15734114#comment-15734114 ] binlijin commented on HBASE-16398: -- Ok, fix it in the HBASE-16398_v3.patch > optimize HRegion computeHDFSBlocksDistribution > -- > > Key: HBASE-16398 > URL: https://issues.apache.org/jira/browse/HBASE-16398 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.0 >Reporter: binlijin >Assignee: binlijin > Fix For: 2.0.0 > > Attachments: HBASE-16398.patch, HBASE-16398_v2.patch, > HBASE-16398_v3.patch, LocatedBlockStatusComparison.java > > > First i assume there is no reference and link in a region family's directory. > Without the patch to computeHDFSBlocksDistribution for a region family, there > is 1+2*N rpc call, N is hfile numbers, The first rpc call is to > DistributedFileSystem#listStatus to get hfiles, for every hfile there is two > rpc call DistributedFileSystem#getFileStatus(path) and then > DistributedFileSystem#getFileBlockLocations(status, start, length). > With the patch to computeHDFSBlocksDistribution for a region family, there is > 2 rpc call, they are DistributedFileSystem#getFileStatus(path) and > DistributedFileSystem#listLocatedStatus(final Path p, final PathFilter > filter). > So if there is at least one hfile, with the patch, the rpc call will less. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17107) FN info should be cleaned up on region/table cleanup
[ https://issues.apache.org/jira/browse/HBASE-17107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvel Thirumoolan updated HBASE-17107: - Status: Patch Available (was: Open) > FN info should be cleaned up on region/table cleanup > > > Key: HBASE-17107 > URL: https://issues.apache.org/jira/browse/HBASE-17107 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-17107.master.001.patch, HBASE_17107.draft.patch > > > FN info should be cleaned up when table is deleted and when regions are GCed > (i.e. CatalogJanitor). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16398) optimize HRegion computeHDFSBlocksDistribution
[ https://issues.apache.org/jira/browse/HBASE-16398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-16398: - Attachment: HBASE-16398_v3.patch > optimize HRegion computeHDFSBlocksDistribution > -- > > Key: HBASE-16398 > URL: https://issues.apache.org/jira/browse/HBASE-16398 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.0 >Reporter: binlijin >Assignee: binlijin > Fix For: 2.0.0 > > Attachments: HBASE-16398.patch, HBASE-16398_v2.patch, > HBASE-16398_v3.patch, LocatedBlockStatusComparison.java > > > First i assume there is no reference and link in a region family's directory. > Without the patch to computeHDFSBlocksDistribution for a region family, there > is 1+2*N rpc call, N is hfile numbers, The first rpc call is to > DistributedFileSystem#listStatus to get hfiles, for every hfile there is two > rpc call DistributedFileSystem#getFileStatus(path) and then > DistributedFileSystem#getFileBlockLocations(status, start, length). > With the patch to computeHDFSBlocksDistribution for a region family, there is > 2 rpc call, they are DistributedFileSystem#getFileStatus(path) and > DistributedFileSystem#listLocatedStatus(final Path p, final PathFilter > filter). > So if there is at least one hfile, with the patch, the rpc call will less. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17107) FN info should be cleaned up on region/table cleanup
[ https://issues.apache.org/jira/browse/HBASE-17107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvel Thirumoolan updated HBASE-17107: - Attachment: HBASE-17107.master.001.patch > FN info should be cleaned up on region/table cleanup > > > Key: HBASE-17107 > URL: https://issues.apache.org/jira/browse/HBASE-17107 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-17107.master.001.patch, HBASE_17107.draft.patch > > > FN info should be cleaned up when table is deleted and when regions are GCed > (i.e. CatalogJanitor). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15734105#comment-15734105 ] Vladimir Rodionov commented on HBASE-14123: --- [~saint@gmail.com] you want me to copy - paste 48 comments from RB and 10 JIRAs descriptions here? Sure, I can do this. No problem. > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Fix For: 2.0.0 > > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, > 14123-master.v21.txt, 14123-master.v24.txt, 14123-master.v25.txt, > 14123-master.v27.txt, 14123-master.v28.txt, 14123-master.v29.full.txt, > 14123-master.v3.txt, 14123-master.v30.txt, 14123-master.v31.txt, > 14123-master.v32.txt, 14123-master.v33.txt, 14123-master.v34.txt, > 14123-master.v35.txt, 14123-master.v36.txt, 14123-master.v37.txt, > 14123-master.v38.txt, 14123-master.v5.txt, 14123-master.v6.txt, > 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, > 14123.master.v39.patch, 14123.master.v40.patch, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15734043#comment-15734043 ] stack commented on HBASE-14123: --- I am supposed to click through a bunch of JIRA numbers to elicit what has been changed? What you are doing is putting on the reviewer the burden of chasing down what was done when you are the one with the knowledge. Please summarize in one place in a manner that is terse but complete and that does not require me clicking about (Links to issues to substantiate claimed fixes are good but I shouldn't have to click through and figure my own summary per item). > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Fix For: 2.0.0 > > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, > 14123-master.v21.txt, 14123-master.v24.txt, 14123-master.v25.txt, > 14123-master.v27.txt, 14123-master.v28.txt, 14123-master.v29.full.txt, > 14123-master.v3.txt, 14123-master.v30.txt, 14123-master.v31.txt, > 14123-master.v32.txt, 14123-master.v33.txt, 14123-master.v34.txt, > 14123-master.v35.txt, 14123-master.v36.txt, 14123-master.v37.txt, > 14123-master.v38.txt, 14123-master.v5.txt, 14123-master.v6.txt, > 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, > 14123.master.v39.patch, 14123.master.v40.patch, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16204) Makefile for running tests using make instead of BUCK
[ https://issues.apache.org/jira/browse/HBASE-16204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15734031#comment-15734031 ] Sudeep Sunthankar commented on HBASE-16204: --- Not required. Please let me know how to delete it and I will do the needful. I couldn't find a way to remove it. > Makefile for running tests using make instead of BUCK > - > > Key: HBASE-16204 > URL: https://issues.apache.org/jira/browse/HBASE-16204 > Project: HBase > Issue Type: Sub-task >Reporter: Sudeep Sunthankar >Assignee: Sudeep Sunthankar > Attachments: HBASE-16204.HBASE-14850.v1.patch > > > This patch consists of addition of Makefile.tests.mk. Currently, make check > runs tests using BUCK. This makefile will compile all the sources written in > *-test.cc and core/simple-client.cc and run them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15734022#comment-15734022 ] Vladimir Rodionov commented on HBASE-14123: --- Addressed most of your comments on RB : https://reviews.apache.org/r/52748/ (see Nov 17, 18) as well as the following JIRAs: HBASE-17134 HBASE-17135 HBASE-17136 HBASE-17146 HBASE-17147 HBASE-17151 HBASE-17152 HBASE-17154 HBASE-17155 HBASE-17156 These tickets addresses your comments posted in this JIRA - not on RB. Will it work for you, [~stack]? > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Fix For: 2.0.0 > > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, > 14123-master.v21.txt, 14123-master.v24.txt, 14123-master.v25.txt, > 14123-master.v27.txt, 14123-master.v28.txt, 14123-master.v29.full.txt, > 14123-master.v3.txt, 14123-master.v30.txt, 14123-master.v31.txt, > 14123-master.v32.txt, 14123-master.v33.txt, 14123-master.v34.txt, > 14123-master.v35.txt, 14123-master.v36.txt, 14123-master.v37.txt, > 14123-master.v38.txt, 14123-master.v5.txt, 14123-master.v6.txt, > 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, > 14123.master.v39.patch, 14123.master.v40.patch, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17280) Add mechanism to control hbase cleaner behavior
[ https://issues.apache.org/jira/browse/HBASE-17280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Jadhav updated HBASE-17280: Attachment: HBASE-17280.branch-1.2.patch HBASE-17280.branch-2.0.patch > Add mechanism to control hbase cleaner behavior > --- > > Key: HBASE-17280 > URL: https://issues.apache.org/jira/browse/HBASE-17280 > Project: HBase > Issue Type: Improvement > Components: Client, hbase, shell >Affects Versions: 2.0.0, 1.2.0 >Reporter: Ajay Jadhav >Priority: Minor > Fix For: 2.0.0, 1.2.0 > > Attachments: HBASE-17280.branch-1.2.patch, > HBASE-17280.branch-2.0.patch > > > Cleaner is used to get rid of archived HFiles and old WALs in HBase. > In the case of heavy workload, cleaner can affect query performance by > creating a lot of connections to perform costly reads/ writes against > underlying filesystem. > This patch allows the user to control HBase cleaner behavior by providing > shell commands to enable/ disable and manually run it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17280) Add mechanism to control hbase cleaner behavior
Ajay Jadhav created HBASE-17280: --- Summary: Add mechanism to control hbase cleaner behavior Key: HBASE-17280 URL: https://issues.apache.org/jira/browse/HBASE-17280 Project: HBase Issue Type: Improvement Components: Client, hbase, shell Affects Versions: 1.2.0, 2.0.0 Reporter: Ajay Jadhav Priority: Minor Fix For: 2.0.0, 1.2.0 Cleaner is used to get rid of archived HFiles and old WALs in HBase. In the case of heavy workload, cleaner can affect query performance by creating a lot of connections to perform costly reads/ writes against underlying filesystem. This patch allows the user to control HBase cleaner behavior by providing shell commands to enable/ disable and manually run it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15756) Pluggable RpcServer
[ https://issues.apache.org/jira/browse/HBASE-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-15756: - Attachment: MultiResponder.master.patch MultiResponder.branch-1.patch > Pluggable RpcServer > --- > > Key: HBASE-15756 > URL: https://issues.apache.org/jira/browse/HBASE-15756 > Project: HBase > Issue Type: Improvement > Components: Performance, rpc >Reporter: binlijin >Assignee: binlijin >Priority: Critical > Fix For: 2.0.0 > > Attachments: MultiResponder.branch-1.patch, > MultiResponder.master.patch, Netty4RpcServer_forperf.patch, > NettyRpcServer.patch, NettyRpcServer_forperf.patch, > PooledByteBufAllocator.patch, PooledByteBufAllocator2.patch, gc.png, > gets.png, gets.png, idle.png, patched.vs.patched_and_cached.vs.no_patch.png, > queue.png > > > Current we use a simple RpcServer, and can not configure and use other > implementation.This issue is to make the RpcServer pluggable, so we can make > other implementation for example netty rpc server. Patch will upload laterly -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17100) Implement Chore to sync FN info from Master to RegionServers
[
https://issues.apache.org/jira/browse/HBASE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733946#comment-15733946
]
Hadoop QA commented on HBASE-17100:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
1s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 40s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 98m 53s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 137m 50s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12842441/HBASE-17100.master.001.patch
|
| JIRA Issue | HBASE-17100 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 290156ba3073 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / b3ae87b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4848/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4848/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Implement Chore to sync FN info from Master to RegionServers
>
>
> Key: HBASE-17100
> URL: https://issues.apache.org/jira/browse/HBASE-17100
> Project: HBase
> Issue Type: Sub-task
>Reporter: Thiruvel Thirumoolan
>Assignee: Thiruvel Thirumoolan
> Fix For: 2.0.0
>
> Attachments: HBASE-17100.master.001.patc
[jira] [Updated] (HBASE-16888) Avoid unnecessary tags copy in Append
[
https://issues.apache.org/jira/browse/HBASE-16888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-16888:
--
Attachment: HBASE-16888.v1.patch
TestAcidGuarantees pass locally.
retry
> Avoid unnecessary tags copy in Append
> -
>
> Key: HBASE-16888
> URL: https://issues.apache.org/jira/browse/HBASE-16888
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16888.v0.patch, HBASE-16888.v1.patch,
> HBASE-16888.v1.patch
>
>
> a) If the delta has tags and the mutation doesn’t apply the TTL, we shouldn’t
> create the TagRewriteCell.
> {noformat}
> List tags = TagUtil.carryForwardTags(delta);
> long ts = now;
> Cell newCell = null;
> byte [] row = mutation.getRow();
> if (currentValue != null) {
> ...
> } else {
> // Append's KeyValue.Type==Put and ts==HConstants.LATEST_TIMESTAMP
> CellUtil.updateLatestStamp(delta, now);
> newCell = delta;
> tags = TagUtil.carryForwardTTLTag(tags, mutation.getTTL());
> if (tags != null) {
> newCell = CellUtil.createCell(delta, tags);
> }
> }
> {noformat}
> b) If the cell has tags, the ShareableMemoryTagRewriteCell will make
> duplicate copy of tags.
> {noformat}
> Cell clonedBaseCell = ((ShareableMemory) this.cell).cloneToCell();
> return new TagRewriteCell(clonedBaseCell, this.tags);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16888) Avoid unnecessary tags copy in Append
[
https://issues.apache.org/jira/browse/HBASE-16888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-16888:
--
Status: Patch Available (was: Open)
> Avoid unnecessary tags copy in Append
> -
>
> Key: HBASE-16888
> URL: https://issues.apache.org/jira/browse/HBASE-16888
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16888.v0.patch, HBASE-16888.v1.patch,
> HBASE-16888.v1.patch
>
>
> a) If the delta has tags and the mutation doesn’t apply the TTL, we shouldn’t
> create the TagRewriteCell.
> {noformat}
> List tags = TagUtil.carryForwardTags(delta);
> long ts = now;
> Cell newCell = null;
> byte [] row = mutation.getRow();
> if (currentValue != null) {
> ...
> } else {
> // Append's KeyValue.Type==Put and ts==HConstants.LATEST_TIMESTAMP
> CellUtil.updateLatestStamp(delta, now);
> newCell = delta;
> tags = TagUtil.carryForwardTTLTag(tags, mutation.getTTL());
> if (tags != null) {
> newCell = CellUtil.createCell(delta, tags);
> }
> }
> {noformat}
> b) If the cell has tags, the ShareableMemoryTagRewriteCell will make
> duplicate copy of tags.
> {noformat}
> Cell clonedBaseCell = ((ShareableMemory) this.cell).cloneToCell();
> return new TagRewriteCell(clonedBaseCell, this.tags);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16888) Avoid unnecessary tags copy in Append
[
https://issues.apache.org/jira/browse/HBASE-16888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-16888:
--
Status: Open (was: Patch Available)
> Avoid unnecessary tags copy in Append
> -
>
> Key: HBASE-16888
> URL: https://issues.apache.org/jira/browse/HBASE-16888
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16888.v0.patch, HBASE-16888.v1.patch
>
>
> a) If the delta has tags and the mutation doesn’t apply the TTL, we shouldn’t
> create the TagRewriteCell.
> {noformat}
> List tags = TagUtil.carryForwardTags(delta);
> long ts = now;
> Cell newCell = null;
> byte [] row = mutation.getRow();
> if (currentValue != null) {
> ...
> } else {
> // Append's KeyValue.Type==Put and ts==HConstants.LATEST_TIMESTAMP
> CellUtil.updateLatestStamp(delta, now);
> newCell = delta;
> tags = TagUtil.carryForwardTTLTag(tags, mutation.getTTL());
> if (tags != null) {
> newCell = CellUtil.createCell(delta, tags);
> }
> }
> {noformat}
> b) If the cell has tags, the ShareableMemoryTagRewriteCell will make
> duplicate copy of tags.
> {noformat}
> Cell clonedBaseCell = ((ShareableMemory) this.cell).cloneToCell();
> return new TagRewriteCell(clonedBaseCell, this.tags);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733930#comment-15733930 ] stack commented on HBASE-14123: --- That is link into the middle of an issue. Usual practice is a tidy description of what has been addressed from review, how, and what has not. The description is posted on rb, here, and best, as part of the patch itself. Comparing two 1MB megapatches for what has been addressed from a review up in RB and JIRA by making some sort of reconciliation between code, JIRA, and RB is beyond my liimited abilities. > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Fix For: 2.0.0 > > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, > 14123-master.v21.txt, 14123-master.v24.txt, 14123-master.v25.txt, > 14123-master.v27.txt, 14123-master.v28.txt, 14123-master.v29.full.txt, > 14123-master.v3.txt, 14123-master.v30.txt, 14123-master.v31.txt, > 14123-master.v32.txt, 14123-master.v33.txt, 14123-master.v34.txt, > 14123-master.v35.txt, 14123-master.v36.txt, 14123-master.v37.txt, > 14123-master.v38.txt, 14123-master.v5.txt, 14123-master.v6.txt, > 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, > 14123.master.v39.patch, 14123.master.v40.patch, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733895#comment-15733895 ] Vladimir Rodionov edited comment on HBASE-14123 at 12/9/16 1:00 AM: [~saint@gmail.com], changes are described here: https://issues.apache.org/jira/browse/HBASE-14123?focusedCommentId=15727077&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15727077 was (Author: vrodionov): https://issues.apache.org/jira/browse/HBASE-14123?focusedCommentId=15727077&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15727077 > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Fix For: 2.0.0 > > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, > 14123-master.v21.txt, 14123-master.v24.txt, 14123-master.v25.txt, > 14123-master.v27.txt, 14123-master.v28.txt, 14123-master.v29.full.txt, > 14123-master.v3.txt, 14123-master.v30.txt, 14123-master.v31.txt, > 14123-master.v32.txt, 14123-master.v33.txt, 14123-master.v34.txt, > 14123-master.v35.txt, 14123-master.v36.txt, 14123-master.v37.txt, > 14123-master.v38.txt, 14123-master.v5.txt, 14123-master.v6.txt, > 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, > 14123.master.v39.patch, 14123.master.v40.patch, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733895#comment-15733895 ] Vladimir Rodionov commented on HBASE-14123: --- https://issues.apache.org/jira/browse/HBASE-14123?focusedCommentId=15727077&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15727077 > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Fix For: 2.0.0 > > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, > 14123-master.v21.txt, 14123-master.v24.txt, 14123-master.v25.txt, > 14123-master.v27.txt, 14123-master.v28.txt, 14123-master.v29.full.txt, > 14123-master.v3.txt, 14123-master.v30.txt, 14123-master.v31.txt, > 14123-master.v32.txt, 14123-master.v33.txt, 14123-master.v34.txt, > 14123-master.v35.txt, 14123-master.v36.txt, 14123-master.v37.txt, > 14123-master.v38.txt, 14123-master.v5.txt, 14123-master.v6.txt, > 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, > 14123.master.v39.patch, 14123.master.v40.patch, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733852#comment-15733852 ] stack commented on HBASE-14123: --- You want to describe what has been changed since the last megapatch [~vrodionov]? > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Fix For: 2.0.0 > > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, > 14123-master.v21.txt, 14123-master.v24.txt, 14123-master.v25.txt, > 14123-master.v27.txt, 14123-master.v28.txt, 14123-master.v29.full.txt, > 14123-master.v3.txt, 14123-master.v30.txt, 14123-master.v31.txt, > 14123-master.v32.txt, 14123-master.v33.txt, 14123-master.v34.txt, > 14123-master.v35.txt, 14123-master.v36.txt, 14123-master.v37.txt, > 14123-master.v38.txt, 14123-master.v5.txt, 14123-master.v6.txt, > 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, > 14123.master.v39.patch, 14123.master.v40.patch, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733843#comment-15733843 ] Vladimir Rodionov commented on HBASE-14123: --- [~saint@gmail.com], new mega patch is ready for your review. > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Fix For: 2.0.0 > > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, > 14123-master.v21.txt, 14123-master.v24.txt, 14123-master.v25.txt, > 14123-master.v27.txt, 14123-master.v28.txt, 14123-master.v29.full.txt, > 14123-master.v3.txt, 14123-master.v30.txt, 14123-master.v31.txt, > 14123-master.v32.txt, 14123-master.v33.txt, 14123-master.v34.txt, > 14123-master.v35.txt, 14123-master.v36.txt, 14123-master.v37.txt, > 14123-master.v38.txt, 14123-master.v5.txt, 14123-master.v6.txt, > 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, > 14123.master.v39.patch, 14123.master.v40.patch, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14417: --- Attachment: 14417-tbl-ext.v11.txt In patch v11, postBulkLoadHFile() would return false for hasLoaded if the write to hbase:backup table doesn't go through. This would trigger retry mechanism in LoadIncrementalHFiles. > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: 14417-tbl-ext.v10.txt, 14417-tbl-ext.v11.txt, > 14417-tbl-ext.v9.txt, 14417.v1.txt, 14417.v11.txt, 14417.v13.txt, > 14417.v2.txt, 14417.v21.txt, 14417.v23.txt, 14417.v24.txt, 14417.v25.txt, > 14417.v6.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-9774) Provide a way for coprocessors to register and report custom metrics
[ https://issues.apache.org/jira/browse/HBASE-9774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar reassigned HBASE-9774: Assignee: Enis Soztutar > Provide a way for coprocessors to register and report custom metrics > > > Key: HBASE-9774 > URL: https://issues.apache.org/jira/browse/HBASE-9774 > Project: HBase > Issue Type: New Feature > Components: Coprocessors, metrics >Reporter: Gary Helmling >Assignee: Enis Soztutar > Attachments: hbase-9774_v1.patch > > > It would help provide better visibility into what coprocessors are doing if > we provided a way for coprocessors to export their own metrics. The general > idea is to: > * extend access to the HBase "metrics bus" down into the coprocessor > environments > * coprocessors can then register and increment custom metrics > * coprocessor metrics are then reported along with all others through normal > mechanisms -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9774) Provide a way for coprocessors to register and report custom metrics
[ https://issues.apache.org/jira/browse/HBASE-9774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-9774: - Status: Patch Available (was: Open) > Provide a way for coprocessors to register and report custom metrics > > > Key: HBASE-9774 > URL: https://issues.apache.org/jira/browse/HBASE-9774 > Project: HBase > Issue Type: New Feature > Components: Coprocessors, metrics >Reporter: Gary Helmling > Attachments: hbase-9774_v1.patch > > > It would help provide better visibility into what coprocessors are doing if > we provided a way for coprocessors to export their own metrics. The general > idea is to: > * extend access to the HBase "metrics bus" down into the coprocessor > environments > * coprocessors can then register and increment custom metrics > * coprocessor metrics are then reported along with all others through normal > mechanisms -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17257) Add column-aliasing capability to hbase-client
[ https://issues.apache.org/jira/browse/HBASE-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733794#comment-15733794 ] Daniel Vimont commented on HBASE-17257: --- Working through an issue brought to my attention by the failures in my first patch submission. After I get that worked out, will post complete code to Review Board! > Add column-aliasing capability to hbase-client > -- > > Key: HBASE-17257 > URL: https://issues.apache.org/jira/browse/HBASE-17257 > Project: HBase > Issue Type: New Feature > Components: Client >Affects Versions: 2.0.0 >Reporter: Daniel Vimont >Assignee: Daniel Vimont > Labels: features > Attachments: HBASE-17257.patch > > > Column aliasing will provide the option for a 1, 2, or 4 byte alias value to > be stored in each cell of an "alias enabled" column-family, in place of the > full-length column-qualifier. Aliasing is intended to operate completely > invisibly to the end-user developer, with absolutely no "awareness" of > aliasing required to be coded into a front-end application. No new public > hbase-client interfaces are to be introduced, and only a few new public > methods should need to be added to existing interfaces, primarily to allow an > administrator to designate that a new column-family is to be alias-enabled by > setting its aliasSize attribute to 1, 2, or 4. > To facilitate such functionality, new subclasses of HTable, > BufferedMutatorImpl, and HTableMultiplexer are to be provided. The overriding > methods of these new subclasses will invoke methods of the new AliasManager > class to facilitate qualifier-to-alias conversions (for user-submitted Gets, > Scans, and Mutations) and alias-to-qualifier conversions (for Results > returned from HBase) for any Table that has one or more alias-enabled column > families. All conversion logic will be encapsulated in the new AliasManager > class, and all qualifier-to-alias mappings will be persisted in a new > aliasMappingTable in a new, reserved namespace. > An informal polling of HBase users at HBaseCon East and at the > Strata/Hadoop-World conference in Sept. 2016 showed that Column Aliasing > could be a popular enhancement to standard HBase functionality, due to the > fact that full column-qualifiers are stored in each cell, and reducing this > qualifier storage requirement down to 1, 2, or 4 bytes per cell could prove > beneficial in terms of reduced storage and bandwidth needs. Aliasing is > intended chiefly for column-families which are of the "narrow and tall" > variety (i.e., that are designed to use relatively few distinct > column-qualifiers throughout a large number of rows, throughout the lifespan > of the column-family). A column-family that is set up with an alias-size of 1 > byte can contain up to 255 unique column-qualifiers; a 2 byte alias-size > allows for up to 65,535 unique column-qualifiers; and a 4 byte alias-size > allows for up to 4,294,967,295 unique column-qualifiers. > Fuller specifications will be entered into the comments section below. Note > that it may well not be viable to add aliasing support in the new "async" > classes that appear to be currently under development. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9774) Provide a way for coprocessors to register and report custom metrics
[ https://issues.apache.org/jira/browse/HBASE-9774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-9774: - Attachment: hbase-9774_v1.patch Attaching patch from review. See RB for more context. > Provide a way for coprocessors to register and report custom metrics > > > Key: HBASE-9774 > URL: https://issues.apache.org/jira/browse/HBASE-9774 > Project: HBase > Issue Type: New Feature > Components: Coprocessors, metrics >Reporter: Gary Helmling > Attachments: hbase-9774_v1.patch > > > It would help provide better visibility into what coprocessors are doing if > we provided a way for coprocessors to export their own metrics. The general > idea is to: > * extend access to the HBase "metrics bus" down into the coprocessor > environments > * coprocessors can then register and increment custom metrics > * coprocessor metrics are then reported along with all others through normal > mechanisms -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17276) Reduce log spam from WrongRegionException in large multi()'s
[
https://issues.apache.org/jira/browse/HBASE-17276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733683#comment-15733683
]
Sean Busbey commented on HBASE-17276:
-
{code}
- LOG.warn("Batch Mutation did not pass sanity check", fsce);
+ final String msg = "Batch Mutation did not pass sanity check. ";
+ if (observedExceptions.hasSeenFailedSanityCheck()) {
+LOG.warn(msg + " " + fsce.getMessage());
+ } else {
+LOG.warn(msg, fsce);
+observedExceptions.sawFailedSanityCheck();
+ }
{code}
nit: unlike the other messages, this one repeats the space both at the end of
{{msg}} and in the string concat in the LOG.
> Reduce log spam from WrongRegionException in large multi()'s
>
>
> Key: HBASE-17276
> URL: https://issues.apache.org/jira/browse/HBASE-17276
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17276.001.patch
>
>
> The following spam drives me up a wall in the regionserver log:
> {noformat}
> 2016-12-05 05:53:05,085 WARN
> [RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=16020]
> regionserver.HRegion: Batch mutation had a row that does not belong to this
> region
> org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out
> of range for doMiniBatchMutation on HRegion
> IntegrationTestReplicationSinkRestart,L\xCC\xCC\xCC\xCC\xCC\xCC\xC8,1480916713541.caab3310166699287b54b72b35b29431.,
> startKey='L\xCC\xCC\xCC\xCC\xCC\xCC\xC8',
> getEndKey()='Y\x99\x99\x99\x99\x99\x99\x94',
> row='\x0C\xD2\xA5\xA3\x99\xC7\xE0Q!\x15^\xA6\x90\x1E\xA3\xAD'
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:5211)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkAndPrepareMutation(HRegion.java:3879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3040)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2933)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2875)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:717)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:679)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2056)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32303)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2141)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
> 2016-12-05 05:53:05,086 WARN
> [RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=16020]
> regionserver.HRegion: Batch mutation had a row that does not belong to this
> region
> org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out
> of range for doMiniBatchMutation on HRegion
> IntegrationTestReplicationSinkRestart,L\xCC\xCC\xCC\xCC\xCC\xCC\xC8,1480916713541.caab3310166699287b54b72b35b29431.,
> startKey='L\xCC\xCC\xCC\xCC\xCC\xCC\xC8',
> getEndKey()='Y\x99\x99\x99\x99\x99\x99\x94',
> row='\x0E\xE7\xFA[\x8D\x93;\xF4\xC7F\xF9\x85\x84\x85\xF3\x0E'
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:5211)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkAndPrepareMutation(HRegion.java:3879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3040)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2933)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2875)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:717)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:679)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2056)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32303)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2141)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
> 2016-12-05 05:
[jira] [Updated] (HBASE-17100) Implement Chore to sync FN info from Master to RegionServers
[ https://issues.apache.org/jira/browse/HBASE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvel Thirumoolan updated HBASE-17100: - Status: Patch Available (was: Open) > Implement Chore to sync FN info from Master to RegionServers > > > Key: HBASE-17100 > URL: https://issues.apache.org/jira/browse/HBASE-17100 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-17100.master.001.patch, HBASE_17100_draft.patch > > > Master will have a repair chore which will periodically sync fn information > from master to all the region servers. This will protect against rpc failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17100) Implement Chore to sync FN info from Master to RegionServers
[ https://issues.apache.org/jira/browse/HBASE-17100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvel Thirumoolan updated HBASE-17100: - Attachment: HBASE-17100.master.001.patch > Implement Chore to sync FN info from Master to RegionServers > > > Key: HBASE-17100 > URL: https://issues.apache.org/jira/browse/HBASE-17100 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-17100.master.001.patch, HBASE_17100_draft.patch > > > Master will have a repair chore which will periodically sync fn information > from master to all the region servers. This will protect against rpc failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14417) Incremental backup and bulk loading
[
https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733635#comment-15733635
]
Ted Yu commented on HBASE-14417:
The final filename is available here in HRegion#bulkLoadHFiles() :
{code}
Path commitedStoreFile = store.bulkLoadHFile(finalPath, seqId);
{code}
If we add one more hook above which records final filename in hbase:backup
table, we still depend on postBulkLoadHFile() hook to write final filename one
more time (with state of completion) - because bulk load event persistence
(done in finally block) may fail. Meaning BackupHFileCleaner wouldn't have
enough information whether the bulk load succeeded by simply checking the
existence of store file(s) in region directory:
{code}
// write a bulk load event when not all hfiles are loaded
try {
WALProtos.BulkLoadDescriptor loadDescriptor =
ProtobufUtil.toBulkLoadDescriptor(
this.getRegionInfo().getTable(),
{code}
> Incremental backup and bulk loading
> ---
>
> Key: HBASE-14417
> URL: https://issues.apache.org/jira/browse/HBASE-14417
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Ted Yu
>Priority: Critical
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: 14417-tbl-ext.v10.txt, 14417-tbl-ext.v9.txt,
> 14417.v1.txt, 14417.v11.txt, 14417.v13.txt, 14417.v2.txt, 14417.v21.txt,
> 14417.v23.txt, 14417.v24.txt, 14417.v25.txt, 14417.v6.txt
>
>
> Currently, incremental backup is based on WAL files. Bulk data loading
> bypasses WALs for obvious reasons, breaking incremental backups. The only way
> to continue backups after bulk loading is to create new full backup of a
> table. This may not be feasible for customers who do bulk loading regularly
> (say, every day).
> Google doc for design:
> https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17276) Reduce log spam from WrongRegionException in large multi()'s
[
https://issues.apache.org/jira/browse/HBASE-17276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733543#comment-15733543
]
Josh Elser commented on HBASE-17276:
Maybe [~enis] or [~busbey] have some thoughts here? :)
> Reduce log spam from WrongRegionException in large multi()'s
>
>
> Key: HBASE-17276
> URL: https://issues.apache.org/jira/browse/HBASE-17276
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17276.001.patch
>
>
> The following spam drives me up a wall in the regionserver log:
> {noformat}
> 2016-12-05 05:53:05,085 WARN
> [RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=16020]
> regionserver.HRegion: Batch mutation had a row that does not belong to this
> region
> org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out
> of range for doMiniBatchMutation on HRegion
> IntegrationTestReplicationSinkRestart,L\xCC\xCC\xCC\xCC\xCC\xCC\xC8,1480916713541.caab3310166699287b54b72b35b29431.,
> startKey='L\xCC\xCC\xCC\xCC\xCC\xCC\xC8',
> getEndKey()='Y\x99\x99\x99\x99\x99\x99\x94',
> row='\x0C\xD2\xA5\xA3\x99\xC7\xE0Q!\x15^\xA6\x90\x1E\xA3\xAD'
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:5211)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkAndPrepareMutation(HRegion.java:3879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3040)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2933)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2875)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:717)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:679)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2056)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32303)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2141)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
> 2016-12-05 05:53:05,086 WARN
> [RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=16020]
> regionserver.HRegion: Batch mutation had a row that does not belong to this
> region
> org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out
> of range for doMiniBatchMutation on HRegion
> IntegrationTestReplicationSinkRestart,L\xCC\xCC\xCC\xCC\xCC\xCC\xC8,1480916713541.caab3310166699287b54b72b35b29431.,
> startKey='L\xCC\xCC\xCC\xCC\xCC\xCC\xC8',
> getEndKey()='Y\x99\x99\x99\x99\x99\x99\x94',
> row='\x0E\xE7\xFA[\x8D\x93;\xF4\xC7F\xF9\x85\x84\x85\xF3\x0E'
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:5211)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkAndPrepareMutation(HRegion.java:3879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3040)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2933)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2875)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:717)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:679)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2056)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32303)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2141)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
> 2016-12-05 05:53:05,087 WARN
> [RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=16020]
> regionserver.HRegion: Batch mutation had a row that does not belong to this
> region
> org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row out
> of range for doMiniBatchMutation on HRegion
> IntegrationTestReplicationSinkRestart,L\xCC\xCC\xCC\xCC\xCC\xCC\xC8,1480916713541.caab3310166699287b54b72b35b29431.,
> startKey='L\xCC\xCC\xCC\x
[jira] [Commented] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733183#comment-15733183 ] Ted Yu commented on HBASE-14417: The sequence Id for bulk loaded hfile is generated inside HRegion#bulkLoadHFiles(). Coming out of bulkLoadHFiles() call, postBulkLoadHFile() hook is called with actual file names. > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: 14417-tbl-ext.v10.txt, 14417-tbl-ext.v9.txt, > 14417.v1.txt, 14417.v11.txt, 14417.v13.txt, 14417.v2.txt, 14417.v21.txt, > 14417.v23.txt, 14417.v24.txt, 14417.v25.txt, 14417.v6.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16894) Create more than 1 split per region, generalize HBASE-12590
[
https://issues.apache.org/jira/browse/HBASE-16894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733167#comment-15733167
]
Yi Liang commented on HBASE-16894:
--
[~enis], [~esteban],
I have attach the code for [~enis] requirement, you can review it at
https://reviews.apache.org/r/54512/
and also attach the implementation doc and some question in the attached file.
Could you guys take a look. Thanks
> Create more than 1 split per region, generalize HBASE-12590
> ---
>
> Key: HBASE-16894
> URL: https://issues.apache.org/jira/browse/HBASE-16894
> Project: HBase
> Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Yi Liang
> Attachments: ImplementaionAndSomeQuestion.docx
>
>
> A common request from users is to be able to better control how many map
> tasks are created per region. Right now, it is always 1 region = 1 input
> split = 1 map task. Same goes for Spark since it uses the TIF. With region
> sizes as large as 50 GBs, it is desirable to be able to create more than 1
> split per region.
> HBASE-12590 adds a config property for MR jobs to be able to handle skew in
> region sizes. The algorithm is roughly:
> {code}
> If (region size >= average size*ratio) : cut the region into two MR input
> splits
> If (average size <= region size < average size*ratio) : one region as one MR
> input split
> If (sum of several continuous regions size < average size * ratio): combine
> these regions into one MR input split.
> {code}
> Although we can set data skew ratio to be 0.5 or something to abuse
> HBASE-12590 into creating more than 1 split task per region, it is not ideal.
> But there is no way to create more with the patch as it is. For example we
> cannot create more than 2 tasks per region.
> If we want to fix this properly, we should extend the approach in
> HBASE-12590, and make it so that the client can specify the desired num of
> mappers, or desired split size, and the TIF generates the splits based on the
> current region sizes very similar to the algorithm in HBASE-12590, but a more
> generic way. This also would eliminate the hand tuning of data skew ratio.
> We also can think about the guidepost approach that Phoenix has in the stats
> table which is used for exactly this purpose. Right now, the region can be
> split into powers of two assuming uniform distribution within the region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16894) Create more than 1 split per region, generalize HBASE-12590
[
https://issues.apache.org/jira/browse/HBASE-16894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liang updated HBASE-16894:
-
Attachment: ImplementaionAndSomeQuestion.docx
> Create more than 1 split per region, generalize HBASE-12590
> ---
>
> Key: HBASE-16894
> URL: https://issues.apache.org/jira/browse/HBASE-16894
> Project: HBase
> Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Yi Liang
> Attachments: ImplementaionAndSomeQuestion.docx
>
>
> A common request from users is to be able to better control how many map
> tasks are created per region. Right now, it is always 1 region = 1 input
> split = 1 map task. Same goes for Spark since it uses the TIF. With region
> sizes as large as 50 GBs, it is desirable to be able to create more than 1
> split per region.
> HBASE-12590 adds a config property for MR jobs to be able to handle skew in
> region sizes. The algorithm is roughly:
> {code}
> If (region size >= average size*ratio) : cut the region into two MR input
> splits
> If (average size <= region size < average size*ratio) : one region as one MR
> input split
> If (sum of several continuous regions size < average size * ratio): combine
> these regions into one MR input split.
> {code}
> Although we can set data skew ratio to be 0.5 or something to abuse
> HBASE-12590 into creating more than 1 split task per region, it is not ideal.
> But there is no way to create more with the patch as it is. For example we
> cannot create more than 2 tasks per region.
> If we want to fix this properly, we should extend the approach in
> HBASE-12590, and make it so that the client can specify the desired num of
> mappers, or desired split size, and the TIF generates the splits based on the
> current region sizes very similar to the algorithm in HBASE-12590, but a more
> generic way. This also would eliminate the hand tuning of data skew ratio.
> We also can think about the guidepost approach that Phoenix has in the stats
> table which is used for exactly this purpose. Right now, the region can be
> split into powers of two assuming uniform distribution within the region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13300) Fix casing in getTimeStamp() and setTimestamp() for Mutations
[ https://issues.apache.org/jira/browse/HBASE-13300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733057#comment-15733057 ] Enis Soztutar commented on HBASE-13300: --- If you can please list all the declared method signatures, it will help us decide. If it is only Delete, I would rather fix it there. > Fix casing in getTimeStamp() and setTimestamp() for Mutations > - > > Key: HBASE-13300 > URL: https://issues.apache.org/jira/browse/HBASE-13300 > Project: HBase > Issue Type: Bug > Components: API >Affects Versions: 1.0.0 >Reporter: Lars George >Assignee: Jan Hentschel > Attachments: HBASE-13300.master.001.patch, > HBASE-13300.master.002.patch > > > For some reason we have two ways of writing this method. It should be > consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16888) Avoid unnecessary tags copy in Append
[
https://issues.apache.org/jira/browse/HBASE-16888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733012#comment-15733012
]
Hadoop QA commented on HBASE-16888:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 59s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 43s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 50s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 42s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 27s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 139m 55s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12842367/HBASE-16888.v1.patch |
| JIRA Issue | HBASE-16888 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 69c71c1a3086 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / b3ae87b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4847/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4847/testReport/ |
| modules | C: hbase-common hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4847/console |
| Powered by | Apache Yetus 0.3.0
[jira] [Commented] (HBASE-16947) Some improvements for DumpReplicationQueues tool
[ https://issues.apache.org/jira/browse/HBASE-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732814#comment-15732814 ] Hudson commented on HBASE-16947: FAILURE: Integrated in Jenkins build HBase-0.98-on-Hadoop-1.1 #1295 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1295/]) HBASE-16947 Some improvements for DumpReplicationQueues tool (apurtell: rev 52566bb0325b9f9ac38450ae04c7a9e5892a493d) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesClientZKImpl.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesZKImpl.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/DumpReplicationQueues.java > Some improvements for DumpReplicationQueues tool > > > Key: HBASE-16947 > URL: https://issues.apache.org/jira/browse/HBASE-16947 > Project: HBase > Issue Type: Improvement > Components: Operability, Replication >Affects Versions: 2.0.0, 1.4.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0, 1.4.0, 1.3.1, 0.98.24 > > Attachments: HBASE-16947-branch-1.patch, HBASE-16947-branch-1.patch, > HBASE-16947-branch-1.patch, HBASE-16947-v1.patch, HBASE-16947.patch > > > Recently we met too many replication WALs problem in our production cluster. > We need the DumpReplicationQueues tool to analyze the replication queues info > in zookeeper. So I backport HBASE-16450 to our branch based 0.98 and did some > improvements for it. > 1. Show the dead regionservers under replication/rs znode. When there are too > many WALs under znode, it can't be atomic transferred to new rs znode. So the > dead rs znode will be leaved on zookeeper. > 2. Make a summary about all the queues that belong to peer has been deleted. > 3. Aggregate all regionservers' size of replication queue. Now the > regionserver report ReplicationLoad to master, but there were not a aggregate > metrics for replication. > 4. Show how many WALs which can not found on hdfs. But the reason (WAL Not > Found) need more time to dig. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16450) Shell tool to dump replication queues
[ https://issues.apache.org/jira/browse/HBASE-16450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732813#comment-15732813 ] Hudson commented on HBASE-16450: FAILURE: Integrated in Jenkins build HBase-0.98-on-Hadoop-1.1 #1295 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1295/]) HBASE-16450 Shell tool to dump replication queues (apurtell: rev 20ee64642f858fe5fdb2bc32811752a85f502aca) * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/io/WALLink.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/DumpReplicationQueues.java > Shell tool to dump replication queues > -- > > Key: HBASE-16450 > URL: https://issues.apache.org/jira/browse/HBASE-16450 > Project: HBase > Issue Type: Improvement > Components: Operability, Replication >Affects Versions: 2.0.0, 1.3.0, 1.1.5, 1.2.2 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez > Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.24 > > Attachments: HBASE-16450.branch-1.001.patch, > HBASE-16450.branch-1.002.patch, HBASE-16450.master.001.patch, > HBASE-16450.master.002.patch, HBASE-16450.master.003.patch > > > Currently there is no way to dump list of the configured queues and the > replication queues when replication is enabled. Unfortunately the HBase > master only offers an option to dump the whole content of the znodes but not > details on the queues being processed on each RS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16450) Shell tool to dump replication queues
[ https://issues.apache.org/jira/browse/HBASE-16450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732757#comment-15732757 ] Hudson commented on HBASE-16450: SUCCESS: Integrated in Jenkins build HBase-0.98-matrix #424 (See [https://builds.apache.org/job/HBase-0.98-matrix/424/]) HBASE-16450 Shell tool to dump replication queues (apurtell: rev 20ee64642f858fe5fdb2bc32811752a85f502aca) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/DumpReplicationQueues.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/io/WALLink.java > Shell tool to dump replication queues > -- > > Key: HBASE-16450 > URL: https://issues.apache.org/jira/browse/HBASE-16450 > Project: HBase > Issue Type: Improvement > Components: Operability, Replication >Affects Versions: 2.0.0, 1.3.0, 1.1.5, 1.2.2 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez > Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.24 > > Attachments: HBASE-16450.branch-1.001.patch, > HBASE-16450.branch-1.002.patch, HBASE-16450.master.001.patch, > HBASE-16450.master.002.patch, HBASE-16450.master.003.patch > > > Currently there is no way to dump list of the configured queues and the > replication queues when replication is enabled. Unfortunately the HBase > master only offers an option to dump the whole content of the znodes but not > details on the queues being processed on each RS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16947) Some improvements for DumpReplicationQueues tool
[ https://issues.apache.org/jira/browse/HBASE-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732758#comment-15732758 ] Hudson commented on HBASE-16947: SUCCESS: Integrated in Jenkins build HBase-0.98-matrix #424 (See [https://builds.apache.org/job/HBase-0.98-matrix/424/]) HBASE-16947 Some improvements for DumpReplicationQueues tool (apurtell: rev 52566bb0325b9f9ac38450ae04c7a9e5892a493d) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesClientZKImpl.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/DumpReplicationQueues.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesZKImpl.java > Some improvements for DumpReplicationQueues tool > > > Key: HBASE-16947 > URL: https://issues.apache.org/jira/browse/HBASE-16947 > Project: HBase > Issue Type: Improvement > Components: Operability, Replication >Affects Versions: 2.0.0, 1.4.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0, 1.4.0, 1.3.1, 0.98.24 > > Attachments: HBASE-16947-branch-1.patch, HBASE-16947-branch-1.patch, > HBASE-16947-branch-1.patch, HBASE-16947-v1.patch, HBASE-16947.patch > > > Recently we met too many replication WALs problem in our production cluster. > We need the DumpReplicationQueues tool to analyze the replication queues info > in zookeeper. So I backport HBASE-16450 to our branch based 0.98 and did some > improvements for it. > 1. Show the dead regionservers under replication/rs znode. When there are too > many WALs under znode, it can't be atomic transferred to new rs znode. So the > dead rs znode will be leaved on zookeeper. > 2. Make a summary about all the queues that belong to peer has been deleted. > 3. Aggregate all regionservers' size of replication queue. Now the > regionserver report ReplicationLoad to master, but there were not a aggregate > metrics for replication. > 4. Show how many WALs which can not found on hdfs. But the reason (WAL Not > Found) need more time to dig. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17277) Allow alternate BufferedMutator implementation
[ https://issues.apache.org/jira/browse/HBASE-17277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732712#comment-15732712 ] stack commented on HBASE-17277: --- bq. This means normally the alternate BM impl has to extend the existing one right? I should change it from private. Will see what implementators want to do ([~jrottinghuis] and [~vrushalic]) first. bq. Can we allow some sort of a callback impl or when the BM tries and fails to flush the mutations in buffered? The impl of such a callback can work with the spooling? We would allow some way to clear off the buffer also then. Just random thoughts... Yeah. This sounds like the way to go. AP will change. AP is too complicated. Exposing a few points of transition seems way to go. > Allow alternate BufferedMutator implementation > -- > > Key: HBASE-17277 > URL: https://issues.apache.org/jira/browse/HBASE-17277 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: stack > Attachments: HBASE-17277.master.001.patch, > HBASE-17277.master.002.patch > > > Allow being able to supply alternate BufferedMutator implementation. For > example, the parent issue would like to spool writes to the filesystem if > hbase is down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16940) Address review of "Backup/Restore (HBASE-7912, HBASE-14030, HBASE-14123) mega patch" posted on RB
[ https://issues.apache.org/jira/browse/HBASE-16940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732650#comment-15732650 ] Ted Yu commented on HBASE-16940: Please fix the remaining findbugs warnings on HBASE-14123 and update the addendum. Thanks > Address review of "Backup/Restore (HBASE-7912, HBASE-14030, HBASE-14123) mega > patch" posted on RB > -- > > Key: HBASE-16940 > URL: https://issues.apache.org/jira/browse/HBASE-16940 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-16940-v1.patch, HBASE-16940-v2.patch, > HBASE-16940.HBASE-7912.add.patch, HBASE-16940.HBASE-7912.v2.patch, > HBASE-16940.addendum, HBASE-16940.addendum.2 > > > Review 52748 remaining issues. > https://reviews.apache.org/r/52748 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16888) Avoid unnecessary tags copy in Append
[
https://issues.apache.org/jira/browse/HBASE-16888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-16888:
--
Attachment: HBASE-16888.v1.patch
> Avoid unnecessary tags copy in Append
> -
>
> Key: HBASE-16888
> URL: https://issues.apache.org/jira/browse/HBASE-16888
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16888.v0.patch, HBASE-16888.v1.patch
>
>
> a) If the delta has tags and the mutation doesn’t apply the TTL, we shouldn’t
> create the TagRewriteCell.
> {noformat}
> List tags = TagUtil.carryForwardTags(delta);
> long ts = now;
> Cell newCell = null;
> byte [] row = mutation.getRow();
> if (currentValue != null) {
> ...
> } else {
> // Append's KeyValue.Type==Put and ts==HConstants.LATEST_TIMESTAMP
> CellUtil.updateLatestStamp(delta, now);
> newCell = delta;
> tags = TagUtil.carryForwardTTLTag(tags, mutation.getTTL());
> if (tags != null) {
> newCell = CellUtil.createCell(delta, tags);
> }
> }
> {noformat}
> b) If the cell has tags, the ShareableMemoryTagRewriteCell will make
> duplicate copy of tags.
> {noformat}
> Cell clonedBaseCell = ((ShareableMemory) this.cell).cloneToCell();
> return new TagRewriteCell(clonedBaseCell, this.tags);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16888) Avoid unnecessary tags copy in Append
[
https://issues.apache.org/jira/browse/HBASE-16888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-16888:
--
Status: Patch Available (was: Open)
> Avoid unnecessary tags copy in Append
> -
>
> Key: HBASE-16888
> URL: https://issues.apache.org/jira/browse/HBASE-16888
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16888.v0.patch, HBASE-16888.v1.patch
>
>
> a) If the delta has tags and the mutation doesn’t apply the TTL, we shouldn’t
> create the TagRewriteCell.
> {noformat}
> List tags = TagUtil.carryForwardTags(delta);
> long ts = now;
> Cell newCell = null;
> byte [] row = mutation.getRow();
> if (currentValue != null) {
> ...
> } else {
> // Append's KeyValue.Type==Put and ts==HConstants.LATEST_TIMESTAMP
> CellUtil.updateLatestStamp(delta, now);
> newCell = delta;
> tags = TagUtil.carryForwardTTLTag(tags, mutation.getTTL());
> if (tags != null) {
> newCell = CellUtil.createCell(delta, tags);
> }
> }
> {noformat}
> b) If the cell has tags, the ShareableMemoryTagRewriteCell will make
> duplicate copy of tags.
> {noformat}
> Cell clonedBaseCell = ((ShareableMemory) this.cell).cloneToCell();
> return new TagRewriteCell(clonedBaseCell, this.tags);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16888) Avoid unnecessary tags copy in Append
[
https://issues.apache.org/jira/browse/HBASE-16888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-16888:
--
Status: Open (was: Patch Available)
> Avoid unnecessary tags copy in Append
> -
>
> Key: HBASE-16888
> URL: https://issues.apache.org/jira/browse/HBASE-16888
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16888.v0.patch, HBASE-16888.v1.patch
>
>
> a) If the delta has tags and the mutation doesn’t apply the TTL, we shouldn’t
> create the TagRewriteCell.
> {noformat}
> List tags = TagUtil.carryForwardTags(delta);
> long ts = now;
> Cell newCell = null;
> byte [] row = mutation.getRow();
> if (currentValue != null) {
> ...
> } else {
> // Append's KeyValue.Type==Put and ts==HConstants.LATEST_TIMESTAMP
> CellUtil.updateLatestStamp(delta, now);
> newCell = delta;
> tags = TagUtil.carryForwardTTLTag(tags, mutation.getTTL());
> if (tags != null) {
> newCell = CellUtil.createCell(delta, tags);
> }
> }
> {noformat}
> b) If the cell has tags, the ShareableMemoryTagRewriteCell will make
> duplicate copy of tags.
> {noformat}
> Cell clonedBaseCell = ((ShareableMemory) this.cell).cloneToCell();
> return new TagRewriteCell(clonedBaseCell, this.tags);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732531#comment-15732531 ] Ted Yu commented on HBASE-14417: More response to Vlad's review comment w.r.t. fault tolerance in bulk load. When bulk load fails midway, the user should provide complete set of hfiles again because the staging directory is not exposed to end users. With this in mind, the benefit of using another hook (prior to postBulkLoadHFile()) to persist location of bulk loaded hfiles is minimal - since in subsequent bulk load attempt(s), the same set of (source) hfiles would be loaded again. Another factor is that the more writes to hbase:backup table, the higher the chance of getting (write) failure. One optimization we can do in the future is to combine writes (performed in postBulkLoadHFile()) from several regions on the same region server, provided that these writes are sufficiently close (300 ms apart, e.g.). The completion of bulk load on a single region server is determined by the slowest participating region, so this optimization would keep the response time on par with the current implementation (where hbase:backup table is not involed). > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: 14417-tbl-ext.v10.txt, 14417-tbl-ext.v9.txt, > 14417.v1.txt, 14417.v11.txt, 14417.v13.txt, 14417.v2.txt, 14417.v21.txt, > 14417.v23.txt, 14417.v24.txt, 14417.v25.txt, 14417.v6.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17182) Memory leak from openScanner of HBase thrift2
[ https://issues.apache.org/jira/browse/HBASE-17182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732473#comment-15732473 ] Ted Yu commented on HBASE-17182: The test was not run by QA. See HBASE-17271 > Memory leak from openScanner of HBase thrift2 > - > > Key: HBASE-17182 > URL: https://issues.apache.org/jira/browse/HBASE-17182 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Jian Yi >Assignee: Jian Yi > Attachments: 17182-V5.patch, 17182-V6.patch, HBASE-17182-V1.patch, > HBASE-17182-V10.patch, HBASE-17182-V2.patch, HBASE-17182-V3.patch, > HBASE-17182-V4.patch, HBASE-17182-V7.patch, HBASE-17182-V8.patch, > HBASE-17182-V9.patch, > org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler-output.txt, > org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler-output.txt > > > Client call openScanner, but client (coredump or others) not closeScanner, > the scanner will not be removed from scannerMap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17182) Memory leak from openScanner of HBase thrift2
[
https://issues.apache.org/jira/browse/HBASE-17182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17182:
---
Attachment:
org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler-output.txt
Test output for patch v10.
{code}
testLongLivedScan(org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler)
Time elapsed: 4.169 sec <<< ERROR!
org.apache.hadoop.hbase.thrift2.generated.TIOError: Failed 1 action:
hconnection-0x2098d37d closed: 1 time, servers with issues: null,
at
org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler.testLongLivedScan(TestThriftHBaseServiceHandler.java:623)
{code}
> Memory leak from openScanner of HBase thrift2
> -
>
> Key: HBASE-17182
> URL: https://issues.apache.org/jira/browse/HBASE-17182
> Project: HBase
> Issue Type: Bug
> Components: Thrift
>Reporter: Jian Yi
>Assignee: Jian Yi
> Attachments: 17182-V5.patch, 17182-V6.patch, HBASE-17182-V1.patch,
> HBASE-17182-V10.patch, HBASE-17182-V2.patch, HBASE-17182-V3.patch,
> HBASE-17182-V4.patch, HBASE-17182-V7.patch, HBASE-17182-V8.patch,
> HBASE-17182-V9.patch,
> org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler-output.txt,
> org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler-output.txt
>
>
> Client call openScanner, but client (coredump or others) not closeScanner,
> the scanner will not be removed from scannerMap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17257) Add column-aliasing capability to hbase-client
[ https://issues.apache.org/jira/browse/HBASE-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732142#comment-15732142 ] Daniel Vimont commented on HBASE-17257: --- OKAY -- "Review Board" -- I stumbled about and found it. Will see if I can open an account and post... > Add column-aliasing capability to hbase-client > -- > > Key: HBASE-17257 > URL: https://issues.apache.org/jira/browse/HBASE-17257 > Project: HBase > Issue Type: New Feature > Components: Client >Affects Versions: 2.0.0 >Reporter: Daniel Vimont >Assignee: Daniel Vimont > Labels: features > Attachments: HBASE-17257.patch > > > Column aliasing will provide the option for a 1, 2, or 4 byte alias value to > be stored in each cell of an "alias enabled" column-family, in place of the > full-length column-qualifier. Aliasing is intended to operate completely > invisibly to the end-user developer, with absolutely no "awareness" of > aliasing required to be coded into a front-end application. No new public > hbase-client interfaces are to be introduced, and only a few new public > methods should need to be added to existing interfaces, primarily to allow an > administrator to designate that a new column-family is to be alias-enabled by > setting its aliasSize attribute to 1, 2, or 4. > To facilitate such functionality, new subclasses of HTable, > BufferedMutatorImpl, and HTableMultiplexer are to be provided. The overriding > methods of these new subclasses will invoke methods of the new AliasManager > class to facilitate qualifier-to-alias conversions (for user-submitted Gets, > Scans, and Mutations) and alias-to-qualifier conversions (for Results > returned from HBase) for any Table that has one or more alias-enabled column > families. All conversion logic will be encapsulated in the new AliasManager > class, and all qualifier-to-alias mappings will be persisted in a new > aliasMappingTable in a new, reserved namespace. > An informal polling of HBase users at HBaseCon East and at the > Strata/Hadoop-World conference in Sept. 2016 showed that Column Aliasing > could be a popular enhancement to standard HBase functionality, due to the > fact that full column-qualifiers are stored in each cell, and reducing this > qualifier storage requirement down to 1, 2, or 4 bytes per cell could prove > beneficial in terms of reduced storage and bandwidth needs. Aliasing is > intended chiefly for column-families which are of the "narrow and tall" > variety (i.e., that are designed to use relatively few distinct > column-qualifiers throughout a large number of rows, throughout the lifespan > of the column-family). A column-family that is set up with an alias-size of 1 > byte can contain up to 255 unique column-qualifiers; a 2 byte alias-size > allows for up to 65,535 unique column-qualifiers; and a 4 byte alias-size > allows for up to 4,294,967,295 unique column-qualifiers. > Fuller specifications will be entered into the comments section below. Note > that it may well not be viable to add aliasing support in the new "async" > classes that appear to be currently under development. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17182) Memory leak from openScanner of HBase thrift2
[ https://issues.apache.org/jira/browse/HBASE-17182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732075#comment-15732075 ] Jian Yi commented on HBASE-17182: - can you tell how to view the result of test. > Memory leak from openScanner of HBase thrift2 > - > > Key: HBASE-17182 > URL: https://issues.apache.org/jira/browse/HBASE-17182 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Jian Yi >Assignee: Jian Yi > Attachments: 17182-V5.patch, 17182-V6.patch, HBASE-17182-V1.patch, > HBASE-17182-V10.patch, HBASE-17182-V2.patch, HBASE-17182-V3.patch, > HBASE-17182-V4.patch, HBASE-17182-V7.patch, HBASE-17182-V8.patch, > HBASE-17182-V9.patch, > org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler-output.txt > > > Client call openScanner, but client (coredump or others) not closeScanner, > the scanner will not be removed from scannerMap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17182) Memory leak from openScanner of HBase thrift2
[
https://issues.apache.org/jira/browse/HBASE-17182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732032#comment-15732032
]
Hadoop QA commented on HBASE-17182:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
23m 25s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s
{color} | {color:green} hbase-thrift in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 36s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12842332/HBASE-17182-V10.patch
|
| JIRA Issue | HBASE-17182 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 0b7ff3fa1a5c 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / b3ae87b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4845/testReport/ |
| modules | C: hbase-thrift U: hbase-thrift |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4845/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Memory leak from openScanner of HBase thrift2
> -
>
> Key: HBASE-17182
> URL: https://issues.apache.org/jira/browse/HBASE-17182
> Project: HBase
> Issue Type: Bug
> Components: Thrift
>Reporter: Jian Yi
>Assignee: Jian Yi
> Attachments: 17182-V5.patch, 17182-V6.patch, HBASE-17182-V1.patch,
> HBASE-17182-V10.patch, HBASE-17182-V2.patch,
[jira] [Updated] (HBASE-17182) Memory leak from openScanner of HBase thrift2
[ https://issues.apache.org/jira/browse/HBASE-17182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Yi updated HBASE-17182: Attachment: HBASE-17182-V10.patch > Memory leak from openScanner of HBase thrift2 > - > > Key: HBASE-17182 > URL: https://issues.apache.org/jira/browse/HBASE-17182 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Jian Yi >Assignee: Jian Yi > Attachments: 17182-V5.patch, 17182-V6.patch, HBASE-17182-V1.patch, > HBASE-17182-V10.patch, HBASE-17182-V2.patch, HBASE-17182-V3.patch, > HBASE-17182-V4.patch, HBASE-17182-V7.patch, HBASE-17182-V8.patch, > HBASE-17182-V9.patch, > org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler-output.txt > > > Client call openScanner, but client (coredump or others) not closeScanner, > the scanner will not be removed from scannerMap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17182) Memory leak from openScanner of HBase thrift2
[ https://issues.apache.org/jira/browse/HBASE-17182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731911#comment-15731911 ] Jian Yi commented on HBASE-17182: - testLongLivedScan cost 1.047 sec? > Memory leak from openScanner of HBase thrift2 > - > > Key: HBASE-17182 > URL: https://issues.apache.org/jira/browse/HBASE-17182 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Jian Yi >Assignee: Jian Yi > Attachments: 17182-V5.patch, 17182-V6.patch, HBASE-17182-V1.patch, > HBASE-17182-V2.patch, HBASE-17182-V3.patch, HBASE-17182-V4.patch, > HBASE-17182-V7.patch, HBASE-17182-V8.patch, HBASE-17182-V9.patch, > org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler-output.txt > > > Client call openScanner, but client (coredump or others) not closeScanner, > the scanner will not be removed from scannerMap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17182) Memory leak from openScanner of HBase thrift2
[
https://issues.apache.org/jira/browse/HBASE-17182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731881#comment-15731881
]
Jian Yi commented on HBASE-17182:
-
This error is because the scanner was removed, getScanner(scannerId) get
scanner failed and throw TIllegalArgument("Invalid scanner Id").
scanPeriod is too small? Maybe getScannerRows costs too much time more than
scanPeriod. So testLongLivedScan need a larger scanPeriod.
> Memory leak from openScanner of HBase thrift2
> -
>
> Key: HBASE-17182
> URL: https://issues.apache.org/jira/browse/HBASE-17182
> Project: HBase
> Issue Type: Bug
> Components: Thrift
>Reporter: Jian Yi
>Assignee: Jian Yi
> Attachments: 17182-V5.patch, 17182-V6.patch, HBASE-17182-V1.patch,
> HBASE-17182-V2.patch, HBASE-17182-V3.patch, HBASE-17182-V4.patch,
> HBASE-17182-V7.patch, HBASE-17182-V8.patch, HBASE-17182-V9.patch,
> org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler-output.txt
>
>
> Client call openScanner, but client (coredump or others) not closeScanner,
> the scanner will not be removed from scannerMap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17279) TestSerialReplication#testRegionSplit is flaky in branch-1
[
https://issues.apache.org/jira/browse/HBASE-17279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731805#comment-15731805
]
Ted Yu commented on HBASE-17279:
I don't think so.
> TestSerialReplication#testRegionSplit is flaky in branch-1
> --
>
> Key: HBASE-17279
> URL: https://issues.apache.org/jira/browse/HBASE-17279
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Phil Yang
>Priority: Minor
>
> From
> https://builds.apache.org/job/PreCommit-HBASE-Build/4832/artifact/patchprocess/patch-unit-hbase-server.txt
> :
> {code}
> testRegionSplit(org.apache.hadoop.hbase.replication.TestSerialReplication)
> Time elapsed: 24.247 sec <<< FAILURE!
> java.lang.AssertionError: expected:<9> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.hadoop.hbase.replication.TestSerialReplication.testRegionSplit(TestSerialReplication.java:256)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17279) TestSerialReplication#testRegionSplit is flaky in branch-1
[
https://issues.apache.org/jira/browse/HBASE-17279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17279:
---
Summary: TestSerialReplication#testRegionSplit is flaky in branch-1 (was:
TestSerialReplication#testRegionSplit is flay)
> TestSerialReplication#testRegionSplit is flaky in branch-1
> --
>
> Key: HBASE-17279
> URL: https://issues.apache.org/jira/browse/HBASE-17279
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Phil Yang
>Priority: Minor
>
> From
> https://builds.apache.org/job/PreCommit-HBASE-Build/4832/artifact/patchprocess/patch-unit-hbase-server.txt
> :
> {code}
> testRegionSplit(org.apache.hadoop.hbase.replication.TestSerialReplication)
> Time elapsed: 24.247 sec <<< FAILURE!
> java.lang.AssertionError: expected:<9> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.hadoop.hbase.replication.TestSerialReplication.testRegionSplit(TestSerialReplication.java:256)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17138) Backport read-path offheap (HBASE-11425) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731803#comment-15731803 ] Ted Yu commented on HBASE-17138: Considerable testing effort is needed on the final artifact with all the backports. The effort involves: 1. showing the backports have the same level of performance boost as seen within Alibaba 2. rolling restart works starting with the build artifact without the backports. 3. ITBLL works with backports (maybe tricky since we don't know whether ITBLL passes for the tip of branch-1 and this is a moving target) [~haoran] Do you have bandwidth for the above actions ? If the answer is yea, you can start the discussion on dev@hbase to get community opinion. > Backport read-path offheap (HBASE-11425) to branch-1 > > > Key: HBASE-17138 > URL: https://issues.apache.org/jira/browse/HBASE-17138 > Project: HBase > Issue Type: Improvement >Reporter: Yu Li >Assignee: Yu Sun > Attachments: > 0001-fix-EHB-511-Resolve-client-compatibility-issue-introduced-by-offheap-change.patch, > 0001-to-EHB-446-offheap-hfile-format-should-keep-compatible-v3.patch, > 0001-to-EHB-456-Cell-should-be-compatible-with-branch-1.1.2.patch > > > From the > [thread|http://mail-archives.apache.org/mod_mbox/hbase-user/201611.mbox/%3CCAM7-19%2Bn7cEiY4H9iLQ3N9V0NXppOPduZwk-hhgNLEaJfiV3kA%40mail.gmail.com%3E] > of sharing our experience and performance data of read-path offheap usage in > Alibaba search, we could see people are positive to have HBASE-11425 in > branch-1, so I'd like to create a JIRA and move the discussion and decision > making here. > Echoing some comments from the mail thread: > Bryan: > Is the backported patch available anywhere? If it ends up not getting > officially backported to branch-1 due to 2.0 around the corner, some of us > who build our own deploy may want to integrate into our builds > Andrew: > Yes, please, the patches will be useful to the community even if we decide > not to backport into an official 1.x release. > Enis: > I don't see any reason why we cannot backport to branch-1. > Ted: > Opening a JIRA would be fine. This makes it easier for people to obtain the > patch(es) > Nick: > From the DISCUSS thread re: EOL of 1.1, it seems we'll continue to > support 1.x releases for some time... I would guess these will be > maintained until 2.2 at least. Therefore, offheap patches that have seen > production exposure seem like a reasonable candidate for backport, perhaps in > a 1.4 or 1.5 release timeframe. > Anoop: > Because of some compatibility issues, we decide that this will be done in 2.0 > only.. Ya as Andy said, it would be great to share the 1.x backported > patches. > The following is all the jira ids we have back ported: > HBASE-10930 Change Filters and GetClosestRowBeforeTracker to work with Cells > (Ram) > HBASE-13373 Squash HFileReaderV3 together with HFileReaderV2 and > AbstractHFileReader; ditto for Scanners and BlockReader, etc. > HBASE-13429 Remove deprecated seek/reseek methods from HFileScanner. > HBASE-13450 - Purge RawBytescomparator from the writers and readers for > HBASE-10800 (Ram) > HBASE-13501 - Deprecate/Remove getComparator() in HRegionInfo. > HBASE-12048 Remove deprecated APIs from Filter. > HBASE-10800 - Use CellComparator instead of KVComparator (Ram) > HBASE-13679 Change ColumnTracker and SQM to deal with Cell instead of byte[], > int, int. > HBASE-13642 Deprecate RegionObserver#postScannerFilterRow CP hook with > byte[],int,int args in favor of taking Cell arg. > HBASE-13641 Deperecate Filter#filterRowKey(byte[] buffer, int offset, int > length) in favor of filterRowKey(Cell firstRowCell). > HBASE-13827 Delayed scanner close in KeyValueHeap and StoreScanner. > HBASE-13871 Change RegionScannerImpl to deal with Cell instead of byte[], > int, int. > HBASE-11911 Break up tests into more fine grained categories (Alex Newman) > HBASE-12059 Create hbase-annotations module > HBASE-12106 Move test annotations to test artifact (Enis Soztutar) > HBASE-13916 Create MultiByteBuffer an aggregation of ByteBuffers. > HBASE-15679 Assertion on wrong variable in > TestReplicationThrottler#testThrottling > HBASE-13931 Move Unsafe based operations to UnsafeAccess. > HBASE-12345 Unsafe based ByteBuffer Comparator. > HBASE-13998 Remove CellComparator#compareRows(byte[] left, int loffset, int > llength, byte[] right, int roffset, int rlength). > HBASE-13998 Remove CellComparator#compareRows()- Addendum to fix javadoc warn > HBASE-13579 Avoid isCellTTLExpired() for NO-TAG cases (partially backport > this patch) > HBASE-13448 New Cell implementation with cached component offsets/lengths. > HBASE-13387 Add ByteBufferedCell an extension to Cell. > HBASE-13387 Add ByteBufferedCell an extension to Cell - adde
[jira] [Updated] (HBASE-17182) Memory leak from openScanner of HBase thrift2
[
https://issues.apache.org/jira/browse/HBASE-17182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17182:
---
Attachment:
org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler-output.txt
Error based on patch v8:
{code}
testLongLivedScan(org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler)
Time elapsed: 1.047 sec <<< ERROR!
org.apache.hadoop.hbase.thrift2.generated.TIllegalArgument: Invalid scanner Id
at
org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler.testLongLivedScan(TestThriftHBaseServiceHandler.java:642)
{code}
Test output attached.
> Memory leak from openScanner of HBase thrift2
> -
>
> Key: HBASE-17182
> URL: https://issues.apache.org/jira/browse/HBASE-17182
> Project: HBase
> Issue Type: Bug
> Components: Thrift
>Reporter: Jian Yi
>Assignee: Jian Yi
> Attachments: 17182-V5.patch, 17182-V6.patch, HBASE-17182-V1.patch,
> HBASE-17182-V2.patch, HBASE-17182-V3.patch, HBASE-17182-V4.patch,
> HBASE-17182-V7.patch, HBASE-17182-V8.patch, HBASE-17182-V9.patch,
> org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler-output.txt
>
>
> Client call openScanner, but client (coredump or others) not closeScanner,
> the scanner will not be removed from scannerMap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17182) Memory leak from openScanner of HBase thrift2
[
https://issues.apache.org/jira/browse/HBASE-17182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731765#comment-15731765
]
Ted Yu commented on HBASE-17182:
v9 didn't compile:
{code}
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
/testptch/hbase/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftHBaseServiceHandler.java:[153,3]
variable scannerCleanerThread might not have been initialized
{code}
> Memory leak from openScanner of HBase thrift2
> -
>
> Key: HBASE-17182
> URL: https://issues.apache.org/jira/browse/HBASE-17182
> Project: HBase
> Issue Type: Bug
> Components: Thrift
>Reporter: Jian Yi
>Assignee: Jian Yi
> Attachments: 17182-V5.patch, 17182-V6.patch, HBASE-17182-V1.patch,
> HBASE-17182-V2.patch, HBASE-17182-V3.patch, HBASE-17182-V4.patch,
> HBASE-17182-V7.patch, HBASE-17182-V8.patch, HBASE-17182-V9.patch
>
>
> Client call openScanner, but client (coredump or others) not closeScanner,
> the scanner will not be removed from scannerMap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17182) Memory leak from openScanner of HBase thrift2
[
https://issues.apache.org/jira/browse/HBASE-17182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731721#comment-15731721
]
Hadoop QA commented on HBASE-17182:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 33s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 18s
{color} | {color:red} hbase-thrift in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 17s
{color} | {color:red} hbase-thrift in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 17s {color}
| {color:red} hbase-thrift in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 27s
{color} | {color:red} The patch causes 14 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 50s
{color} | {color:red} The patch causes 14 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 13s
{color} | {color:red} The patch causes 14 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 36s
{color} | {color:red} The patch causes 14 errors with Hadoop v2.6.4. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m 1s
{color} | {color:red} The patch causes 14 errors with Hadoop v2.6.5. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 22s
{color} | {color:red} The patch causes 14 errors with Hadoop v2.7.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 9m 47s
{color} | {color:red} The patch causes 14 errors with Hadoop v2.7.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 11m 12s
{color} | {color:red} The patch causes 14 errors with Hadoop v2.7.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 12m 33s
{color} | {color:red} The patch causes 14 errors with Hadoop v3.0.0-alpha1.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 15s
{color} | {color:red} hbase-thrift in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 17s {color}
| {color:red} hbase-thrift in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
6s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 31s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12842319/HBASE-17182-V9.patch |
| JIRA Issue | HBASE-17182 |
| Optional Tests | asflicense javac javadoc unit
[jira] [Commented] (HBASE-17182) Memory leak from openScanner of HBase thrift2
[ https://issues.apache.org/jira/browse/HBASE-17182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731676#comment-15731676 ] Jian Yi commented on HBASE-17182: - Can you send me the error logs? > Memory leak from openScanner of HBase thrift2 > - > > Key: HBASE-17182 > URL: https://issues.apache.org/jira/browse/HBASE-17182 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Jian Yi >Assignee: Jian Yi > Attachments: 17182-V5.patch, 17182-V6.patch, HBASE-17182-V1.patch, > HBASE-17182-V2.patch, HBASE-17182-V3.patch, HBASE-17182-V4.patch, > HBASE-17182-V7.patch, HBASE-17182-V8.patch, HBASE-17182-V9.patch > > > Client call openScanner, but client (coredump or others) not closeScanner, > the scanner will not be removed from scannerMap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17182) Memory leak from openScanner of HBase thrift2
[ https://issues.apache.org/jira/browse/HBASE-17182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731673#comment-15731673 ] Jian Yi commented on HBASE-17182: - Sorry, currently my windows can not access HBase cluster for network policy. I need run TestThriftHBaseServiceHandler on linux to access hbase cluster, but mvn need internet to download dependencies. > Memory leak from openScanner of HBase thrift2 > - > > Key: HBASE-17182 > URL: https://issues.apache.org/jira/browse/HBASE-17182 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Jian Yi >Assignee: Jian Yi > Attachments: 17182-V5.patch, 17182-V6.patch, HBASE-17182-V1.patch, > HBASE-17182-V2.patch, HBASE-17182-V3.patch, HBASE-17182-V4.patch, > HBASE-17182-V7.patch, HBASE-17182-V8.patch, HBASE-17182-V9.patch > > > Client call openScanner, but client (coredump or others) not closeScanner, > the scanner will not be removed from scannerMap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17182) Memory leak from openScanner of HBase thrift2
[ https://issues.apache.org/jira/browse/HBASE-17182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Yi updated HBASE-17182: Attachment: HBASE-17182-V9.patch limit the min value of scanPeriod > Memory leak from openScanner of HBase thrift2 > - > > Key: HBASE-17182 > URL: https://issues.apache.org/jira/browse/HBASE-17182 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Jian Yi >Assignee: Jian Yi > Attachments: 17182-V5.patch, 17182-V6.patch, HBASE-17182-V1.patch, > HBASE-17182-V2.patch, HBASE-17182-V3.patch, HBASE-17182-V4.patch, > HBASE-17182-V7.patch, HBASE-17182-V8.patch, HBASE-17182-V9.patch > > > Client call openScanner, but client (coredump or others) not closeScanner, > the scanner will not be removed from scannerMap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-17138) Backport read-path offheap (HBASE-11425) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731546#comment-15731546 ] Anoop Sam John edited comment on HBASE-17138 at 12/8/16 8:46 AM: - With all these list and analysis, lets start a vote in dev@. Then we can decide based on the voting pattern there. Ya a new branch based on branch-1 might be needed with smaller patches getting in and then finally a merge back to branch-1 was (Author: anoop.hbase): With all these list and analysis, lets start a vote in dev@. Then we can decide based on the voting pattern there. > Backport read-path offheap (HBASE-11425) to branch-1 > > > Key: HBASE-17138 > URL: https://issues.apache.org/jira/browse/HBASE-17138 > Project: HBase > Issue Type: Improvement >Reporter: Yu Li >Assignee: Yu Sun > Attachments: > 0001-fix-EHB-511-Resolve-client-compatibility-issue-introduced-by-offheap-change.patch, > 0001-to-EHB-446-offheap-hfile-format-should-keep-compatible-v3.patch, > 0001-to-EHB-456-Cell-should-be-compatible-with-branch-1.1.2.patch > > > From the > [thread|http://mail-archives.apache.org/mod_mbox/hbase-user/201611.mbox/%3CCAM7-19%2Bn7cEiY4H9iLQ3N9V0NXppOPduZwk-hhgNLEaJfiV3kA%40mail.gmail.com%3E] > of sharing our experience and performance data of read-path offheap usage in > Alibaba search, we could see people are positive to have HBASE-11425 in > branch-1, so I'd like to create a JIRA and move the discussion and decision > making here. > Echoing some comments from the mail thread: > Bryan: > Is the backported patch available anywhere? If it ends up not getting > officially backported to branch-1 due to 2.0 around the corner, some of us > who build our own deploy may want to integrate into our builds > Andrew: > Yes, please, the patches will be useful to the community even if we decide > not to backport into an official 1.x release. > Enis: > I don't see any reason why we cannot backport to branch-1. > Ted: > Opening a JIRA would be fine. This makes it easier for people to obtain the > patch(es) > Nick: > From the DISCUSS thread re: EOL of 1.1, it seems we'll continue to > support 1.x releases for some time... I would guess these will be > maintained until 2.2 at least. Therefore, offheap patches that have seen > production exposure seem like a reasonable candidate for backport, perhaps in > a 1.4 or 1.5 release timeframe. > Anoop: > Because of some compatibility issues, we decide that this will be done in 2.0 > only.. Ya as Andy said, it would be great to share the 1.x backported > patches. > The following is all the jira ids we have back ported: > HBASE-10930 Change Filters and GetClosestRowBeforeTracker to work with Cells > (Ram) > HBASE-13373 Squash HFileReaderV3 together with HFileReaderV2 and > AbstractHFileReader; ditto for Scanners and BlockReader, etc. > HBASE-13429 Remove deprecated seek/reseek methods from HFileScanner. > HBASE-13450 - Purge RawBytescomparator from the writers and readers for > HBASE-10800 (Ram) > HBASE-13501 - Deprecate/Remove getComparator() in HRegionInfo. > HBASE-12048 Remove deprecated APIs from Filter. > HBASE-10800 - Use CellComparator instead of KVComparator (Ram) > HBASE-13679 Change ColumnTracker and SQM to deal with Cell instead of byte[], > int, int. > HBASE-13642 Deprecate RegionObserver#postScannerFilterRow CP hook with > byte[],int,int args in favor of taking Cell arg. > HBASE-13641 Deperecate Filter#filterRowKey(byte[] buffer, int offset, int > length) in favor of filterRowKey(Cell firstRowCell). > HBASE-13827 Delayed scanner close in KeyValueHeap and StoreScanner. > HBASE-13871 Change RegionScannerImpl to deal with Cell instead of byte[], > int, int. > HBASE-11911 Break up tests into more fine grained categories (Alex Newman) > HBASE-12059 Create hbase-annotations module > HBASE-12106 Move test annotations to test artifact (Enis Soztutar) > HBASE-13916 Create MultiByteBuffer an aggregation of ByteBuffers. > HBASE-15679 Assertion on wrong variable in > TestReplicationThrottler#testThrottling > HBASE-13931 Move Unsafe based operations to UnsafeAccess. > HBASE-12345 Unsafe based ByteBuffer Comparator. > HBASE-13998 Remove CellComparator#compareRows(byte[] left, int loffset, int > llength, byte[] right, int roffset, int rlength). > HBASE-13998 Remove CellComparator#compareRows()- Addendum to fix javadoc warn > HBASE-13579 Avoid isCellTTLExpired() for NO-TAG cases (partially backport > this patch) > HBASE-13448 New Cell implementation with cached component offsets/lengths. > HBASE-13387 Add ByteBufferedCell an extension to Cell. > HBASE-13387 Add ByteBufferedCell an extension to Cell - addendum. > HBASE-12650 Move ServerName to hbase-common module (partially backport this > patch) > HBASE-12296 Fil
[jira] [Commented] (HBASE-17138) Backport read-path offheap (HBASE-11425) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731563#comment-15731563 ] Anoop Sam John commented on HBASE-17138: Thanks for the patches.. Ya all make sense.. On Filter, so we have Cell based APIs from 0.96 and deprecated KV based APIs there. But 1.x version contain both. So in branch-1 how is it used now? The core code flow calls cell based new APIs right? And FilterBase def impl of new Cell based APIs try to call KV based old APIs.CPs follow this way. I did not see branch-1 code base. I think ur attached patch is having some issue wrt maintain BC. Separate topic , we can discuss. So the end result is that we can conclude that the backport of the feature is possible with BC maintain. Good .. > Backport read-path offheap (HBASE-11425) to branch-1 > > > Key: HBASE-17138 > URL: https://issues.apache.org/jira/browse/HBASE-17138 > Project: HBase > Issue Type: Improvement >Reporter: Yu Li >Assignee: Yu Sun > Attachments: > 0001-fix-EHB-511-Resolve-client-compatibility-issue-introduced-by-offheap-change.patch, > 0001-to-EHB-446-offheap-hfile-format-should-keep-compatible-v3.patch, > 0001-to-EHB-456-Cell-should-be-compatible-with-branch-1.1.2.patch > > > From the > [thread|http://mail-archives.apache.org/mod_mbox/hbase-user/201611.mbox/%3CCAM7-19%2Bn7cEiY4H9iLQ3N9V0NXppOPduZwk-hhgNLEaJfiV3kA%40mail.gmail.com%3E] > of sharing our experience and performance data of read-path offheap usage in > Alibaba search, we could see people are positive to have HBASE-11425 in > branch-1, so I'd like to create a JIRA and move the discussion and decision > making here. > Echoing some comments from the mail thread: > Bryan: > Is the backported patch available anywhere? If it ends up not getting > officially backported to branch-1 due to 2.0 around the corner, some of us > who build our own deploy may want to integrate into our builds > Andrew: > Yes, please, the patches will be useful to the community even if we decide > not to backport into an official 1.x release. > Enis: > I don't see any reason why we cannot backport to branch-1. > Ted: > Opening a JIRA would be fine. This makes it easier for people to obtain the > patch(es) > Nick: > From the DISCUSS thread re: EOL of 1.1, it seems we'll continue to > support 1.x releases for some time... I would guess these will be > maintained until 2.2 at least. Therefore, offheap patches that have seen > production exposure seem like a reasonable candidate for backport, perhaps in > a 1.4 or 1.5 release timeframe. > Anoop: > Because of some compatibility issues, we decide that this will be done in 2.0 > only.. Ya as Andy said, it would be great to share the 1.x backported > patches. > The following is all the jira ids we have back ported: > HBASE-10930 Change Filters and GetClosestRowBeforeTracker to work with Cells > (Ram) > HBASE-13373 Squash HFileReaderV3 together with HFileReaderV2 and > AbstractHFileReader; ditto for Scanners and BlockReader, etc. > HBASE-13429 Remove deprecated seek/reseek methods from HFileScanner. > HBASE-13450 - Purge RawBytescomparator from the writers and readers for > HBASE-10800 (Ram) > HBASE-13501 - Deprecate/Remove getComparator() in HRegionInfo. > HBASE-12048 Remove deprecated APIs from Filter. > HBASE-10800 - Use CellComparator instead of KVComparator (Ram) > HBASE-13679 Change ColumnTracker and SQM to deal with Cell instead of byte[], > int, int. > HBASE-13642 Deprecate RegionObserver#postScannerFilterRow CP hook with > byte[],int,int args in favor of taking Cell arg. > HBASE-13641 Deperecate Filter#filterRowKey(byte[] buffer, int offset, int > length) in favor of filterRowKey(Cell firstRowCell). > HBASE-13827 Delayed scanner close in KeyValueHeap and StoreScanner. > HBASE-13871 Change RegionScannerImpl to deal with Cell instead of byte[], > int, int. > HBASE-11911 Break up tests into more fine grained categories (Alex Newman) > HBASE-12059 Create hbase-annotations module > HBASE-12106 Move test annotations to test artifact (Enis Soztutar) > HBASE-13916 Create MultiByteBuffer an aggregation of ByteBuffers. > HBASE-15679 Assertion on wrong variable in > TestReplicationThrottler#testThrottling > HBASE-13931 Move Unsafe based operations to UnsafeAccess. > HBASE-12345 Unsafe based ByteBuffer Comparator. > HBASE-13998 Remove CellComparator#compareRows(byte[] left, int loffset, int > llength, byte[] right, int roffset, int rlength). > HBASE-13998 Remove CellComparator#compareRows()- Addendum to fix javadoc warn > HBASE-13579 Avoid isCellTTLExpired() for NO-TAG cases (partially backport > this patch) > HBASE-13448 New Cell implementation with cached component offsets/lengths. > HBASE-13387 Add ByteBufferedCell an extension to Cell. >
[jira] [Commented] (HBASE-15716) HRegion#RegionScannerImpl scannerReadPoints synchronization constrains random read
[ https://issues.apache.org/jira/browse/HBASE-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731545#comment-15731545 ] Yu Li commented on HBASE-15716: --- Ping, this is something worth moving on according to our test result (smile). > HRegion#RegionScannerImpl scannerReadPoints synchronization constrains random > read > -- > > Key: HBASE-15716 > URL: https://issues.apache.org/jira/browse/HBASE-15716 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: stack >Assignee: stack > Attachments: > 15716.implementation.using.ScannerReadPoints.branch-1.patch, > 15716.prune.synchronizations.patch, 15716.prune.synchronizations.v3.patch, > 15716.prune.synchronizations.v4.patch, 15716.prune.synchronizations.v4.patch, > 15716.wip.more_to_be_done.patch, HBASE-15716.branch-1.001.patch, > HBASE-15716.branch-1.002.patch, HBASE-15716.branch-1.003.patch, > HBASE-15716.branch-1.004.patch, HBASE-15716.branch-1.005.patch, > ScannerReadPoints.java, ScannerReadPoints.v2.java, Screen Shot 2016-04-26 at > 2.05.45 PM.png, Screen Shot 2016-04-26 at 2.06.14 PM.png, Screen Shot > 2016-04-26 at 2.07.06 PM.png, Screen Shot 2016-04-26 at 2.25.26 PM.png, > Screen Shot 2016-04-26 at 6.02.29 PM.png, Screen Shot 2016-04-27 at 9.49.35 > AM.png, Screen Shot 2016-06-30 at 9.52.52 PM.png, Screen Shot 2016-06-30 at > 9.54.08 PM.png, TestScannerReadPointDoesntGoBack.java, > TestScannerReadPoints.java, before_after.png, > current-branch-1.vs.NoSynchronization.vs.Patch.png, hits.png, > remove.locks.patch, remove_cslm.patch > > > Here is a [~lhofhansl] special. > When we construct the region scanner, we get our read point and then store it > with the scanner instance in a Region scoped CSLM. This is done under a > synchronize on the CSLM. > This synchronize on a region-scoped Map creating region scanners is the > outstanding point of lock contention according to flight recorder (My work > load is workload c, random reads). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17138) Backport read-path offheap (HBASE-11425) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731546#comment-15731546 ] Anoop Sam John commented on HBASE-17138: With all these list and analysis, lets start a vote in dev@. Then we can decide based on the voting pattern there. > Backport read-path offheap (HBASE-11425) to branch-1 > > > Key: HBASE-17138 > URL: https://issues.apache.org/jira/browse/HBASE-17138 > Project: HBase > Issue Type: Improvement >Reporter: Yu Li >Assignee: Yu Sun > Attachments: > 0001-fix-EHB-511-Resolve-client-compatibility-issue-introduced-by-offheap-change.patch, > 0001-to-EHB-446-offheap-hfile-format-should-keep-compatible-v3.patch, > 0001-to-EHB-456-Cell-should-be-compatible-with-branch-1.1.2.patch > > > From the > [thread|http://mail-archives.apache.org/mod_mbox/hbase-user/201611.mbox/%3CCAM7-19%2Bn7cEiY4H9iLQ3N9V0NXppOPduZwk-hhgNLEaJfiV3kA%40mail.gmail.com%3E] > of sharing our experience and performance data of read-path offheap usage in > Alibaba search, we could see people are positive to have HBASE-11425 in > branch-1, so I'd like to create a JIRA and move the discussion and decision > making here. > Echoing some comments from the mail thread: > Bryan: > Is the backported patch available anywhere? If it ends up not getting > officially backported to branch-1 due to 2.0 around the corner, some of us > who build our own deploy may want to integrate into our builds > Andrew: > Yes, please, the patches will be useful to the community even if we decide > not to backport into an official 1.x release. > Enis: > I don't see any reason why we cannot backport to branch-1. > Ted: > Opening a JIRA would be fine. This makes it easier for people to obtain the > patch(es) > Nick: > From the DISCUSS thread re: EOL of 1.1, it seems we'll continue to > support 1.x releases for some time... I would guess these will be > maintained until 2.2 at least. Therefore, offheap patches that have seen > production exposure seem like a reasonable candidate for backport, perhaps in > a 1.4 or 1.5 release timeframe. > Anoop: > Because of some compatibility issues, we decide that this will be done in 2.0 > only.. Ya as Andy said, it would be great to share the 1.x backported > patches. > The following is all the jira ids we have back ported: > HBASE-10930 Change Filters and GetClosestRowBeforeTracker to work with Cells > (Ram) > HBASE-13373 Squash HFileReaderV3 together with HFileReaderV2 and > AbstractHFileReader; ditto for Scanners and BlockReader, etc. > HBASE-13429 Remove deprecated seek/reseek methods from HFileScanner. > HBASE-13450 - Purge RawBytescomparator from the writers and readers for > HBASE-10800 (Ram) > HBASE-13501 - Deprecate/Remove getComparator() in HRegionInfo. > HBASE-12048 Remove deprecated APIs from Filter. > HBASE-10800 - Use CellComparator instead of KVComparator (Ram) > HBASE-13679 Change ColumnTracker and SQM to deal with Cell instead of byte[], > int, int. > HBASE-13642 Deprecate RegionObserver#postScannerFilterRow CP hook with > byte[],int,int args in favor of taking Cell arg. > HBASE-13641 Deperecate Filter#filterRowKey(byte[] buffer, int offset, int > length) in favor of filterRowKey(Cell firstRowCell). > HBASE-13827 Delayed scanner close in KeyValueHeap and StoreScanner. > HBASE-13871 Change RegionScannerImpl to deal with Cell instead of byte[], > int, int. > HBASE-11911 Break up tests into more fine grained categories (Alex Newman) > HBASE-12059 Create hbase-annotations module > HBASE-12106 Move test annotations to test artifact (Enis Soztutar) > HBASE-13916 Create MultiByteBuffer an aggregation of ByteBuffers. > HBASE-15679 Assertion on wrong variable in > TestReplicationThrottler#testThrottling > HBASE-13931 Move Unsafe based operations to UnsafeAccess. > HBASE-12345 Unsafe based ByteBuffer Comparator. > HBASE-13998 Remove CellComparator#compareRows(byte[] left, int loffset, int > llength, byte[] right, int roffset, int rlength). > HBASE-13998 Remove CellComparator#compareRows()- Addendum to fix javadoc warn > HBASE-13579 Avoid isCellTTLExpired() for NO-TAG cases (partially backport > this patch) > HBASE-13448 New Cell implementation with cached component offsets/lengths. > HBASE-13387 Add ByteBufferedCell an extension to Cell. > HBASE-13387 Add ByteBufferedCell an extension to Cell - addendum. > HBASE-12650 Move ServerName to hbase-common module (partially backport this > patch) > HBASE-12296 Filters should work with ByteBufferedCell. > HBASE-14120 ByteBufferUtils#compareTo small optimization. > HBASE-13510 - Purge ByteBloomFilter (Ram) > HBASE-13451 - Make the HFileBlockIndex blockKeys to Cells so that it could be > easy to use in the CellComparators (Ram) > HBASE-13614 - Avoid temp KeyOnlyKeyValue temp object
[jira] [Commented] (HBASE-17138) Backport read-path offheap (HBASE-11425) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731515#comment-15731515 ] Yu Li commented on HBASE-17138: --- Ok, we get a long list to backport and some new changes to keep compatibility, good start (smile). So what's the better way for next move? Fork and merge? Or simply a big patch here for review? No doubt it's time costing to make sure of compatibility and stability no matter which way to take, but worth the efforts. :-) > Backport read-path offheap (HBASE-11425) to branch-1 > > > Key: HBASE-17138 > URL: https://issues.apache.org/jira/browse/HBASE-17138 > Project: HBase > Issue Type: Improvement >Reporter: Yu Li >Assignee: Yu Sun > Attachments: > 0001-fix-EHB-511-Resolve-client-compatibility-issue-introduced-by-offheap-change.patch, > 0001-to-EHB-446-offheap-hfile-format-should-keep-compatible-v3.patch, > 0001-to-EHB-456-Cell-should-be-compatible-with-branch-1.1.2.patch > > > From the > [thread|http://mail-archives.apache.org/mod_mbox/hbase-user/201611.mbox/%3CCAM7-19%2Bn7cEiY4H9iLQ3N9V0NXppOPduZwk-hhgNLEaJfiV3kA%40mail.gmail.com%3E] > of sharing our experience and performance data of read-path offheap usage in > Alibaba search, we could see people are positive to have HBASE-11425 in > branch-1, so I'd like to create a JIRA and move the discussion and decision > making here. > Echoing some comments from the mail thread: > Bryan: > Is the backported patch available anywhere? If it ends up not getting > officially backported to branch-1 due to 2.0 around the corner, some of us > who build our own deploy may want to integrate into our builds > Andrew: > Yes, please, the patches will be useful to the community even if we decide > not to backport into an official 1.x release. > Enis: > I don't see any reason why we cannot backport to branch-1. > Ted: > Opening a JIRA would be fine. This makes it easier for people to obtain the > patch(es) > Nick: > From the DISCUSS thread re: EOL of 1.1, it seems we'll continue to > support 1.x releases for some time... I would guess these will be > maintained until 2.2 at least. Therefore, offheap patches that have seen > production exposure seem like a reasonable candidate for backport, perhaps in > a 1.4 or 1.5 release timeframe. > Anoop: > Because of some compatibility issues, we decide that this will be done in 2.0 > only.. Ya as Andy said, it would be great to share the 1.x backported > patches. > The following is all the jira ids we have back ported: > HBASE-10930 Change Filters and GetClosestRowBeforeTracker to work with Cells > (Ram) > HBASE-13373 Squash HFileReaderV3 together with HFileReaderV2 and > AbstractHFileReader; ditto for Scanners and BlockReader, etc. > HBASE-13429 Remove deprecated seek/reseek methods from HFileScanner. > HBASE-13450 - Purge RawBytescomparator from the writers and readers for > HBASE-10800 (Ram) > HBASE-13501 - Deprecate/Remove getComparator() in HRegionInfo. > HBASE-12048 Remove deprecated APIs from Filter. > HBASE-10800 - Use CellComparator instead of KVComparator (Ram) > HBASE-13679 Change ColumnTracker and SQM to deal with Cell instead of byte[], > int, int. > HBASE-13642 Deprecate RegionObserver#postScannerFilterRow CP hook with > byte[],int,int args in favor of taking Cell arg. > HBASE-13641 Deperecate Filter#filterRowKey(byte[] buffer, int offset, int > length) in favor of filterRowKey(Cell firstRowCell). > HBASE-13827 Delayed scanner close in KeyValueHeap and StoreScanner. > HBASE-13871 Change RegionScannerImpl to deal with Cell instead of byte[], > int, int. > HBASE-11911 Break up tests into more fine grained categories (Alex Newman) > HBASE-12059 Create hbase-annotations module > HBASE-12106 Move test annotations to test artifact (Enis Soztutar) > HBASE-13916 Create MultiByteBuffer an aggregation of ByteBuffers. > HBASE-15679 Assertion on wrong variable in > TestReplicationThrottler#testThrottling > HBASE-13931 Move Unsafe based operations to UnsafeAccess. > HBASE-12345 Unsafe based ByteBuffer Comparator. > HBASE-13998 Remove CellComparator#compareRows(byte[] left, int loffset, int > llength, byte[] right, int roffset, int rlength). > HBASE-13998 Remove CellComparator#compareRows()- Addendum to fix javadoc warn > HBASE-13579 Avoid isCellTTLExpired() for NO-TAG cases (partially backport > this patch) > HBASE-13448 New Cell implementation with cached component offsets/lengths. > HBASE-13387 Add ByteBufferedCell an extension to Cell. > HBASE-13387 Add ByteBufferedCell an extension to Cell - addendum. > HBASE-12650 Move ServerName to hbase-common module (partially backport this > patch) > HBASE-12296 Filters should work with ByteBufferedCell. > HBASE-14120 ByteBufferUtils#compareTo small optimization. > HBASE-13510 - Purge ByteBloomF
[jira] [Commented] (HBASE-17138) Backport read-path offheap (HBASE-11425) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731493#comment-15731493 ] Yu Sun commented on HBASE-17138: yes, I think so, almost all of patches need to be changed due to merge conflict.so this will need some time if we decide to backport. > Backport read-path offheap (HBASE-11425) to branch-1 > > > Key: HBASE-17138 > URL: https://issues.apache.org/jira/browse/HBASE-17138 > Project: HBase > Issue Type: Improvement >Reporter: Yu Li >Assignee: Yu Sun > Attachments: > 0001-fix-EHB-511-Resolve-client-compatibility-issue-introduced-by-offheap-change.patch, > 0001-to-EHB-446-offheap-hfile-format-should-keep-compatible-v3.patch, > 0001-to-EHB-456-Cell-should-be-compatible-with-branch-1.1.2.patch > > > From the > [thread|http://mail-archives.apache.org/mod_mbox/hbase-user/201611.mbox/%3CCAM7-19%2Bn7cEiY4H9iLQ3N9V0NXppOPduZwk-hhgNLEaJfiV3kA%40mail.gmail.com%3E] > of sharing our experience and performance data of read-path offheap usage in > Alibaba search, we could see people are positive to have HBASE-11425 in > branch-1, so I'd like to create a JIRA and move the discussion and decision > making here. > Echoing some comments from the mail thread: > Bryan: > Is the backported patch available anywhere? If it ends up not getting > officially backported to branch-1 due to 2.0 around the corner, some of us > who build our own deploy may want to integrate into our builds > Andrew: > Yes, please, the patches will be useful to the community even if we decide > not to backport into an official 1.x release. > Enis: > I don't see any reason why we cannot backport to branch-1. > Ted: > Opening a JIRA would be fine. This makes it easier for people to obtain the > patch(es) > Nick: > From the DISCUSS thread re: EOL of 1.1, it seems we'll continue to > support 1.x releases for some time... I would guess these will be > maintained until 2.2 at least. Therefore, offheap patches that have seen > production exposure seem like a reasonable candidate for backport, perhaps in > a 1.4 or 1.5 release timeframe. > Anoop: > Because of some compatibility issues, we decide that this will be done in 2.0 > only.. Ya as Andy said, it would be great to share the 1.x backported > patches. > The following is all the jira ids we have back ported: > HBASE-10930 Change Filters and GetClosestRowBeforeTracker to work with Cells > (Ram) > HBASE-13373 Squash HFileReaderV3 together with HFileReaderV2 and > AbstractHFileReader; ditto for Scanners and BlockReader, etc. > HBASE-13429 Remove deprecated seek/reseek methods from HFileScanner. > HBASE-13450 - Purge RawBytescomparator from the writers and readers for > HBASE-10800 (Ram) > HBASE-13501 - Deprecate/Remove getComparator() in HRegionInfo. > HBASE-12048 Remove deprecated APIs from Filter. > HBASE-10800 - Use CellComparator instead of KVComparator (Ram) > HBASE-13679 Change ColumnTracker and SQM to deal with Cell instead of byte[], > int, int. > HBASE-13642 Deprecate RegionObserver#postScannerFilterRow CP hook with > byte[],int,int args in favor of taking Cell arg. > HBASE-13641 Deperecate Filter#filterRowKey(byte[] buffer, int offset, int > length) in favor of filterRowKey(Cell firstRowCell). > HBASE-13827 Delayed scanner close in KeyValueHeap and StoreScanner. > HBASE-13871 Change RegionScannerImpl to deal with Cell instead of byte[], > int, int. > HBASE-11911 Break up tests into more fine grained categories (Alex Newman) > HBASE-12059 Create hbase-annotations module > HBASE-12106 Move test annotations to test artifact (Enis Soztutar) > HBASE-13916 Create MultiByteBuffer an aggregation of ByteBuffers. > HBASE-15679 Assertion on wrong variable in > TestReplicationThrottler#testThrottling > HBASE-13931 Move Unsafe based operations to UnsafeAccess. > HBASE-12345 Unsafe based ByteBuffer Comparator. > HBASE-13998 Remove CellComparator#compareRows(byte[] left, int loffset, int > llength, byte[] right, int roffset, int rlength). > HBASE-13998 Remove CellComparator#compareRows()- Addendum to fix javadoc warn > HBASE-13579 Avoid isCellTTLExpired() for NO-TAG cases (partially backport > this patch) > HBASE-13448 New Cell implementation with cached component offsets/lengths. > HBASE-13387 Add ByteBufferedCell an extension to Cell. > HBASE-13387 Add ByteBufferedCell an extension to Cell - addendum. > HBASE-12650 Move ServerName to hbase-common module (partially backport this > patch) > HBASE-12296 Filters should work with ByteBufferedCell. > HBASE-14120 ByteBufferUtils#compareTo small optimization. > HBASE-13510 - Purge ByteBloomFilter (Ram) > HBASE-13451 - Make the HFileBlockIndex blockKeys to Cells so that it could be > easy to use in the CellComparators (Ram) > HBASE-13614 - Avoid temp KeyOnlyKeyValue temp obje
[jira] [Commented] (HBASE-17138) Backport read-path offheap (HBASE-11425) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731485#comment-15731485 ] Yu Sun commented on HBASE-17138: done > Backport read-path offheap (HBASE-11425) to branch-1 > > > Key: HBASE-17138 > URL: https://issues.apache.org/jira/browse/HBASE-17138 > Project: HBase > Issue Type: Improvement >Reporter: Yu Li >Assignee: Yu Sun > Attachments: > 0001-fix-EHB-511-Resolve-client-compatibility-issue-introduced-by-offheap-change.patch, > 0001-to-EHB-446-offheap-hfile-format-should-keep-compatible-v3.patch, > 0001-to-EHB-456-Cell-should-be-compatible-with-branch-1.1.2.patch > > > From the > [thread|http://mail-archives.apache.org/mod_mbox/hbase-user/201611.mbox/%3CCAM7-19%2Bn7cEiY4H9iLQ3N9V0NXppOPduZwk-hhgNLEaJfiV3kA%40mail.gmail.com%3E] > of sharing our experience and performance data of read-path offheap usage in > Alibaba search, we could see people are positive to have HBASE-11425 in > branch-1, so I'd like to create a JIRA and move the discussion and decision > making here. > Echoing some comments from the mail thread: > Bryan: > Is the backported patch available anywhere? If it ends up not getting > officially backported to branch-1 due to 2.0 around the corner, some of us > who build our own deploy may want to integrate into our builds > Andrew: > Yes, please, the patches will be useful to the community even if we decide > not to backport into an official 1.x release. > Enis: > I don't see any reason why we cannot backport to branch-1. > Ted: > Opening a JIRA would be fine. This makes it easier for people to obtain the > patch(es) > Nick: > From the DISCUSS thread re: EOL of 1.1, it seems we'll continue to > support 1.x releases for some time... I would guess these will be > maintained until 2.2 at least. Therefore, offheap patches that have seen > production exposure seem like a reasonable candidate for backport, perhaps in > a 1.4 or 1.5 release timeframe. > Anoop: > Because of some compatibility issues, we decide that this will be done in 2.0 > only.. Ya as Andy said, it would be great to share the 1.x backported > patches. > The following is all the jira ids we have back ported: > HBASE-10930 Change Filters and GetClosestRowBeforeTracker to work with Cells > (Ram) > HBASE-13373 Squash HFileReaderV3 together with HFileReaderV2 and > AbstractHFileReader; ditto for Scanners and BlockReader, etc. > HBASE-13429 Remove deprecated seek/reseek methods from HFileScanner. > HBASE-13450 - Purge RawBytescomparator from the writers and readers for > HBASE-10800 (Ram) > HBASE-13501 - Deprecate/Remove getComparator() in HRegionInfo. > HBASE-12048 Remove deprecated APIs from Filter. > HBASE-10800 - Use CellComparator instead of KVComparator (Ram) > HBASE-13679 Change ColumnTracker and SQM to deal with Cell instead of byte[], > int, int. > HBASE-13642 Deprecate RegionObserver#postScannerFilterRow CP hook with > byte[],int,int args in favor of taking Cell arg. > HBASE-13641 Deperecate Filter#filterRowKey(byte[] buffer, int offset, int > length) in favor of filterRowKey(Cell firstRowCell). > HBASE-13827 Delayed scanner close in KeyValueHeap and StoreScanner. > HBASE-13871 Change RegionScannerImpl to deal with Cell instead of byte[], > int, int. > HBASE-11911 Break up tests into more fine grained categories (Alex Newman) > HBASE-12059 Create hbase-annotations module > HBASE-12106 Move test annotations to test artifact (Enis Soztutar) > HBASE-13916 Create MultiByteBuffer an aggregation of ByteBuffers. > HBASE-15679 Assertion on wrong variable in > TestReplicationThrottler#testThrottling > HBASE-13931 Move Unsafe based operations to UnsafeAccess. > HBASE-12345 Unsafe based ByteBuffer Comparator. > HBASE-13998 Remove CellComparator#compareRows(byte[] left, int loffset, int > llength, byte[] right, int roffset, int rlength). > HBASE-13998 Remove CellComparator#compareRows()- Addendum to fix javadoc warn > HBASE-13579 Avoid isCellTTLExpired() for NO-TAG cases (partially backport > this patch) > HBASE-13448 New Cell implementation with cached component offsets/lengths. > HBASE-13387 Add ByteBufferedCell an extension to Cell. > HBASE-13387 Add ByteBufferedCell an extension to Cell - addendum. > HBASE-12650 Move ServerName to hbase-common module (partially backport this > patch) > HBASE-12296 Filters should work with ByteBufferedCell. > HBASE-14120 ByteBufferUtils#compareTo small optimization. > HBASE-13510 - Purge ByteBloomFilter (Ram) > HBASE-13451 - Make the HFileBlockIndex blockKeys to Cells so that it could be > easy to use in the CellComparators (Ram) > HBASE-13614 - Avoid temp KeyOnlyKeyValue temp objects creations in read hot > path (Ram) > HBASE-13939 - Make HFileReaderImpl.getFirstKeyInBlock() to return a Cell (Ram) > HBASE-1
