[jira] [Commented] (HBASE-17723) ClientAsyncPrefetchScanner may end prematurely when the size of the cache is one
[
https://issues.apache.org/jira/browse/HBASE-17723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906938#comment-15906938
]
Hadoop QA commented on HBASE-17723:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 24s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 12s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 20s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 100m 29s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 148m 49s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12857549/HBASE-17723.v1.patch |
| JIRA Issue | HBASE-17723 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 1fbc036235e4 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 31bc94a |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6070/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6070/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6070/console |
| Powered by | Apache Yetus 0.3.0
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906946#comment-15906946 ] ramkrishna.s.vasudevan commented on HBASE-16438: Hi All I have updated the patch in RB. There are few things I have added in the RB. To reiterate what is there in RB 1) The ChunkCell is now an extension of BytebufferKV. The problem is that since BBKV already has seqId in it even if we serailize the seqId in the buffer then that ref is of no use. So we may need new Cell impl 2) The chunkId is now added only to the beginning of each chunk and not in every cell. 3) As far as this patch is concerned I feel we can focus on chunkCreation and using this ChunkCell. But adding the seqId inside the buffer and using that buffer has to be done while changing 1) and also integrate it with CellChunkMap flow. 4) This patch also handles clearing the map inside ChunkCreator when there is no chunkPool and we have MSLAB only. > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, > MemstoreChunkCell_trunk.patch > > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14161) Add hbase-spark integration tests to IT jenkins job
[ https://issues.apache.org/jira/browse/HBASE-14161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906948#comment-15906948 ] Sean Busbey commented on HBASE-14161: - I don't see the spark ITs in [the trunk IT matrix|https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk-IT/]. [~jinghe] could you add a note about how this is a duplicate? > Add hbase-spark integration tests to IT jenkins job > --- > > Key: HBASE-14161 > URL: https://issues.apache.org/jira/browse/HBASE-14161 > Project: HBase > Issue Type: Task > Components: build >Reporter: Sean Busbey > Fix For: 2.0.0 > > > expand the set of ITs we run to include the new hbase-spark tests. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17778) Remove the testing code in the AsyncRequestFutureImpl
[
https://issues.apache.org/jira/browse/HBASE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906951#comment-15906951
]
Hadoop QA commented on HBASE-17778:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 57s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 21s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 28s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12857563/HBASE-17778.v0.patch |
| JIRA Issue | HBASE-17778 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 9e387fbf50bc 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 31bc94a |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6072/testReport/ |
| modules | C: hbase-client U: hbase-client |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6072/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Remove the testing code in the AsyncRequestFutureImpl
> -
>
> Key: HBASE-17778
> URL: https://issues.apache.org/jira/browse/HBASE-17778
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Trivial
> Fix For: 2.0.0
>
> Attachments: HBA
[jira] [Commented] (HBASE-15597) Clean up configuration keys used in hbase-spark module
[ https://issues.apache.org/jira/browse/HBASE-15597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906957#comment-15906957 ] Sean Busbey commented on HBASE-15597: - +1 > Clean up configuration keys used in hbase-spark module > -- > > Key: HBASE-15597 > URL: https://issues.apache.org/jira/browse/HBASE-15597 > Project: HBase > Issue Type: Bug > Components: spark >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Yi Liang >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-15597-v1.patch, HBASE-15597-V2.patch, > HBASE-15597-V3.patch, HBase-15597-V4.patch > > > This should be considered a blocker for backport to branch-1 since it will > impact our compatibility. > The constants we expose in configuration should all start with "hbase". Since > our configurations keys for the spark integration all relate to that system, > the prefix for all configuration keys (excluding those cases where we need to > do something special due to restrictions in how properties are handled by > e.g. spark) should be "hbase.spark". > Before publishing a public api labeled version of our spark integration we > should review all of our configuration keys to make sure they either conform > to the "hbase.spark" prefix or they have a comment documenting why they need > to be otherwise. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17746) TestSimpleRpcScheduler.testCoDelScheduling is broken
[ https://issues.apache.org/jira/browse/HBASE-17746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906974#comment-15906974 ] Hudson commented on HBASE-17746: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2663 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2663/]) HBASE-17746 TestSimpleRpcScheduler.testCoDelScheduling is broken (zghao: rev 31bc94ae6094d02a73d347549e23bcbaff97838f) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestSimpleRpcScheduler.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java > TestSimpleRpcScheduler.testCoDelScheduling is broken > > > Key: HBASE-17746 > URL: https://issues.apache.org/jira/browse/HBASE-17746 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 2.0.0 > Environment: OS: Ubuntu 14.04 > Arch: x86_64 > Linux x 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 > x86_64 x86_64 x86_64 GNU/Linux >Reporter: Anup Halarnkar >Assignee: Chia-Ping Tsai > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17746.branch-1.001.patch, HBASE-17746.v0.patch, > HBASE-17746.v1.patch > > > Command: > export JAVA_TOOL_OPTIONS="-Xmx8G -Xms2G" > mvn clean install -X > Output: > Results : > Failed tests: > TestSimpleRpcScheduler.testCoDelScheduling:469 None of these calls should > have been discarded expected:<0> but was:<15> > Tests run: 1191, Failures: 1, Errors: 0, Skipped: 12 > Output of "TestSimpleRpcScheduler.txt" > --- > Test set: org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler > --- > Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.642 sec <<< > FAILURE! - in org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler > testCoDelScheduling(org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler) Time > elapsed: 0.563 sec <<< FAILURE! > java.lang.AssertionError: None of these calls should have been discarded > expected:<0> but was:<15> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at > org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler.testCoDelScheduling(TestSimpleRpcScheduler.java:469) > -- > Output of last few lines of "TestSimpleRpcScheduler-output.txt" > 2017-03-07 16:09:16,580 INFO [main] hbase.ResourceChecker(172): after: > ipc.TestSimpleRpcScheduler#testCoDelScheduling Thread=5 (was 5), > OpenFileDescriptor=250 (was 250), MaxFileDescriptor=4096 (was 4096), > SystemLoadAverage=607 (was 607), ProcessCount=158 (was 157) - ProcessCount > LEAK? -, AvailableMemoryMB=1734 (was 1762) > 2017-03-07 16:09:16,604 INFO [main] hbase.ResourceChecker(148): before: > ipc.TestSimpleRpcScheduler#testSoftAndHardQueueLimits Thread=5, > OpenFileDescriptor=252, MaxFileDescriptor=4096, SystemLoadAverage=607, > ProcessCount=158, AvailableMemoryMB=1734 > 2017-03-07 16:09:16,722 INFO [main] ipc.RpcExecutor(145): RpcExecutor name > using fifo as call queue; numCallQueues=1; maxQueueLength=5; handlerCount=0 > 2017-03-07 16:09:16,722 DEBUG [main] ipc.RpcExecutor(215): Started > RpcServer.deafult.FPBQ.Fifo.handler=0,queue=0,port=0 > 2017-03-07 16:09:16,839 INFO [main] hbase.ResourceChecker(172): after: > ipc.TestSimpleRpcScheduler#testSoftAndHardQueueLimits Thread=5 (was 5), > OpenFileDescriptor=250 (was 252), MaxFileDescriptor=4096 (was 4096), > SystemLoadAverage=607 (was 607), ProcessCount=158 (was 158), > AvailableMemoryMB=1703 (was 1734) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17758) [RSGROUP] Add shell command to move servers and tables at the same time
[
https://issues.apache.org/jira/browse/HBASE-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guangxu Cheng updated HBASE-17758:
--
Attachment: HBASE-17758-v4.patch
> [RSGROUP] Add shell command to move servers and tables at the same time
> ---
>
> Key: HBASE-17758
> URL: https://issues.apache.org/jira/browse/HBASE-17758
> Project: HBase
> Issue Type: New Feature
> Components: rsgroup
>Affects Versions: 2.0.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
> Attachments: HBASE-17758-v1.patch, HBASE-17758-v2.patch,
> HBASE-17758-v3.patch, HBASE-17758-v4.patch
>
>
> Currently add a new group perform the following steps:
> {code:javascript}
> hbase(main):001:0> add_rsgroup 'mygroup'
> Took 0.3840 seconds
> hbase(main):002:0> move_servers_rsgroup
> 'mygroup',['hbase-rs-01:16030','hbase-rs-02:16030']
> Took 3.5040 seconds
> hbase(main):003:0> move_tables_rsgroup 'mygroup',['example']
> Took 0.2710 seconds
> {code}
> 1. move_servers_rsgroup will unassign all regions on hbase-rs-01 and
> hbase-rs-02
> 2. move_tables_rsgroup will also reassign all the regions of the table
> example.
> This will lead to a large number of regions to migrate and affect the data
> locality.
> However, some regions of the table that are distributed on hbase-rs-01 and
> hbase-rs-02, do not need to be moved.
> So,we need a new shell command *move_servers_tables_rsgroup* which minimizes
> the number of regions needed to move.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17758) [RSGROUP] Add shell command to move servers and tables at the same time
[
https://issues.apache.org/jira/browse/HBASE-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906987#comment-15906987
]
Guangxu Cheng commented on HBASE-17758:
---
The v4 patch fixes failed tests and add server movement fails scenario in
testMoveServersAndTables.
> [RSGROUP] Add shell command to move servers and tables at the same time
> ---
>
> Key: HBASE-17758
> URL: https://issues.apache.org/jira/browse/HBASE-17758
> Project: HBase
> Issue Type: New Feature
> Components: rsgroup
>Affects Versions: 2.0.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
> Attachments: HBASE-17758-v1.patch, HBASE-17758-v2.patch,
> HBASE-17758-v3.patch, HBASE-17758-v4.patch
>
>
> Currently add a new group perform the following steps:
> {code:javascript}
> hbase(main):001:0> add_rsgroup 'mygroup'
> Took 0.3840 seconds
> hbase(main):002:0> move_servers_rsgroup
> 'mygroup',['hbase-rs-01:16030','hbase-rs-02:16030']
> Took 3.5040 seconds
> hbase(main):003:0> move_tables_rsgroup 'mygroup',['example']
> Took 0.2710 seconds
> {code}
> 1. move_servers_rsgroup will unassign all regions on hbase-rs-01 and
> hbase-rs-02
> 2. move_tables_rsgroup will also reassign all the regions of the table
> example.
> This will lead to a large number of regions to migrate and affect the data
> locality.
> However, some regions of the table that are distributed on hbase-rs-01 and
> hbase-rs-02, do not need to be moved.
> So,we need a new shell command *move_servers_tables_rsgroup* which minimizes
> the number of regions needed to move.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17723) ClientAsyncPrefetchScanner may end prematurely when the size of the cache is one
[ https://issues.apache.org/jira/browse/HBASE-17723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17723: --- Attachment: HBASE-17723.v1.patch > ClientAsyncPrefetchScanner may end prematurely when the size of the cache is > one > > > Key: HBASE-17723 > URL: https://issues.apache.org/jira/browse/HBASE-17723 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 > Environment: Ubuntu 14.04 LTS / x86_64 / 16GB RAM >Reporter: Anup Halarnkar >Assignee: Chia-Ping Tsai > Fix For: 2.0.0 > > Attachments: HBASE-17723.v0.patch, HBASE-17723.v1.patch, > HBASE-17723.v1.patch, HBASE-17723.v1.patch, HBASE-17723.v1.patch > > > Command to build and test: > mvn clean install -X > JAVA_TOOL_OPTIONS: -Xmx4G -XX:MaxPermSize=4G -Xms1G > Failure in tests: > Results : > Failed tests: > > TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync:161->testRowsSeenMetric:174->testRowsSeenMetric:204->testMetric:330 > Metric: ROWS_SCANNED Expected: 9 Actual: 8 expected:<9> but was:<8> > > TestRecoveredEdits.testReplayWorksThoughLotsOfFlushing:78->testReplayWorksWithMemoryCompactionPolicy:149->verifyAllEditsMadeItIn:195 > Failed to find > W\xE1X\xDBv\xD8}\x07\xBF\xC1I\xF1\xF5\xB9\x84\xD8/meta:prev/1422491946444/Put/vlen=16/seqid=0 > Tests run: 1761, Failures: 2, Errors: 0, Skipped: 6 > > This is the output for 1st test failure: > --- > Test set: org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > --- > Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.971 sec <<< > FAILURE! - in org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > testRowsSeenMetricWithAsync(org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide) > Time elapsed: 1.134 sec <<< FAILURE! > java.lang.AssertionError: Metric: ROWS_SCANNED Expected: 9 Actual: 8 > expected:<9> but was:<8> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testMetric(TestServerSideScanMetricsFromClientSide.java:330) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetric(TestServerSideScanMetricsFromClientSide.java:204) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetric(TestServerSideScanMetricsFromClientSide.java:174) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync(TestServerSideScanMetricsFromClientSide.java:161) > > This is output of 2nd test failure: > --- > Test set: org.apache.hadoop.hbase.regionserver.TestRecoveredEdits > --- > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 11.911 sec > <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestRecoveredEdits > testReplayWorksThoughLotsOfFlushing(org.apache.hadoop.hbase.regionserver.TestRecoveredEdits) > Time elapsed: 11.871 sec <<< FAILURE! > java.lang.AssertionError: Failed to find > W\xE1X\xDBv\xD8}\x07\xBF\xC1I\xF1\xF5\xB9\x84\xD8/meta:prev/1422491946444/Put/vlen=16/seqid=0 > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.verifyAllEditsMadeItIn(TestRecoveredEdits.java:195) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.testReplayWorksWithMemoryCompactionPolicy(TestRecoveredEdits.java:149) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.testReplayWorksThoughLotsOfFlushing(TestRecoveredEdits.java:78) > Please let me know if you need more information > Thanks, > Anup -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17723) ClientAsyncPrefetchScanner may end prematurely when the size of the cache is one
[ https://issues.apache.org/jira/browse/HBASE-17723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17723: --- Status: Open (was: Patch Available) TestRegionReplicaFailover pass locally and it is unrelated to this patch. Retry > ClientAsyncPrefetchScanner may end prematurely when the size of the cache is > one > > > Key: HBASE-17723 > URL: https://issues.apache.org/jira/browse/HBASE-17723 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 > Environment: Ubuntu 14.04 LTS / x86_64 / 16GB RAM >Reporter: Anup Halarnkar >Assignee: Chia-Ping Tsai > Fix For: 2.0.0 > > Attachments: HBASE-17723.v0.patch, HBASE-17723.v1.patch, > HBASE-17723.v1.patch, HBASE-17723.v1.patch, HBASE-17723.v1.patch > > > Command to build and test: > mvn clean install -X > JAVA_TOOL_OPTIONS: -Xmx4G -XX:MaxPermSize=4G -Xms1G > Failure in tests: > Results : > Failed tests: > > TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync:161->testRowsSeenMetric:174->testRowsSeenMetric:204->testMetric:330 > Metric: ROWS_SCANNED Expected: 9 Actual: 8 expected:<9> but was:<8> > > TestRecoveredEdits.testReplayWorksThoughLotsOfFlushing:78->testReplayWorksWithMemoryCompactionPolicy:149->verifyAllEditsMadeItIn:195 > Failed to find > W\xE1X\xDBv\xD8}\x07\xBF\xC1I\xF1\xF5\xB9\x84\xD8/meta:prev/1422491946444/Put/vlen=16/seqid=0 > Tests run: 1761, Failures: 2, Errors: 0, Skipped: 6 > > This is the output for 1st test failure: > --- > Test set: org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > --- > Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.971 sec <<< > FAILURE! - in org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > testRowsSeenMetricWithAsync(org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide) > Time elapsed: 1.134 sec <<< FAILURE! > java.lang.AssertionError: Metric: ROWS_SCANNED Expected: 9 Actual: 8 > expected:<9> but was:<8> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testMetric(TestServerSideScanMetricsFromClientSide.java:330) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetric(TestServerSideScanMetricsFromClientSide.java:204) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetric(TestServerSideScanMetricsFromClientSide.java:174) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync(TestServerSideScanMetricsFromClientSide.java:161) > > This is output of 2nd test failure: > --- > Test set: org.apache.hadoop.hbase.regionserver.TestRecoveredEdits > --- > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 11.911 sec > <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestRecoveredEdits > testReplayWorksThoughLotsOfFlushing(org.apache.hadoop.hbase.regionserver.TestRecoveredEdits) > Time elapsed: 11.871 sec <<< FAILURE! > java.lang.AssertionError: Failed to find > W\xE1X\xDBv\xD8}\x07\xBF\xC1I\xF1\xF5\xB9\x84\xD8/meta:prev/1422491946444/Put/vlen=16/seqid=0 > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.verifyAllEditsMadeItIn(TestRecoveredEdits.java:195) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.testReplayWorksWithMemoryCompactionPolicy(TestRecoveredEdits.java:149) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.testReplayWorksThoughLotsOfFlushing(TestRecoveredEdits.java:78) > Please let me know if you need more information > Thanks, > Anup -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17723) ClientAsyncPrefetchScanner may end prematurely when the size of the cache is one
[ https://issues.apache.org/jira/browse/HBASE-17723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17723: --- Status: Patch Available (was: Open) > ClientAsyncPrefetchScanner may end prematurely when the size of the cache is > one > > > Key: HBASE-17723 > URL: https://issues.apache.org/jira/browse/HBASE-17723 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 > Environment: Ubuntu 14.04 LTS / x86_64 / 16GB RAM >Reporter: Anup Halarnkar >Assignee: Chia-Ping Tsai > Fix For: 2.0.0 > > Attachments: HBASE-17723.v0.patch, HBASE-17723.v1.patch, > HBASE-17723.v1.patch, HBASE-17723.v1.patch, HBASE-17723.v1.patch > > > Command to build and test: > mvn clean install -X > JAVA_TOOL_OPTIONS: -Xmx4G -XX:MaxPermSize=4G -Xms1G > Failure in tests: > Results : > Failed tests: > > TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync:161->testRowsSeenMetric:174->testRowsSeenMetric:204->testMetric:330 > Metric: ROWS_SCANNED Expected: 9 Actual: 8 expected:<9> but was:<8> > > TestRecoveredEdits.testReplayWorksThoughLotsOfFlushing:78->testReplayWorksWithMemoryCompactionPolicy:149->verifyAllEditsMadeItIn:195 > Failed to find > W\xE1X\xDBv\xD8}\x07\xBF\xC1I\xF1\xF5\xB9\x84\xD8/meta:prev/1422491946444/Put/vlen=16/seqid=0 > Tests run: 1761, Failures: 2, Errors: 0, Skipped: 6 > > This is the output for 1st test failure: > --- > Test set: org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > --- > Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.971 sec <<< > FAILURE! - in org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > testRowsSeenMetricWithAsync(org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide) > Time elapsed: 1.134 sec <<< FAILURE! > java.lang.AssertionError: Metric: ROWS_SCANNED Expected: 9 Actual: 8 > expected:<9> but was:<8> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testMetric(TestServerSideScanMetricsFromClientSide.java:330) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetric(TestServerSideScanMetricsFromClientSide.java:204) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetric(TestServerSideScanMetricsFromClientSide.java:174) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync(TestServerSideScanMetricsFromClientSide.java:161) > > This is output of 2nd test failure: > --- > Test set: org.apache.hadoop.hbase.regionserver.TestRecoveredEdits > --- > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 11.911 sec > <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestRecoveredEdits > testReplayWorksThoughLotsOfFlushing(org.apache.hadoop.hbase.regionserver.TestRecoveredEdits) > Time elapsed: 11.871 sec <<< FAILURE! > java.lang.AssertionError: Failed to find > W\xE1X\xDBv\xD8}\x07\xBF\xC1I\xF1\xF5\xB9\x84\xD8/meta:prev/1422491946444/Put/vlen=16/seqid=0 > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.verifyAllEditsMadeItIn(TestRecoveredEdits.java:195) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.testReplayWorksWithMemoryCompactionPolicy(TestRecoveredEdits.java:149) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.testReplayWorksThoughLotsOfFlushing(TestRecoveredEdits.java:78) > Please let me know if you need more information > Thanks, > Anup -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17778) Remove the testing code in the AsyncRequestFutureImpl
[ https://issues.apache.org/jira/browse/HBASE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17778: --- Status: Open (was: Patch Available) > Remove the testing code in the AsyncRequestFutureImpl > - > > Key: HBASE-17778 > URL: https://issues.apache.org/jira/browse/HBASE-17778 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17778.v0.patch > > > HBASE-16224 left some testing code in the AsyncRequestFutureImpl. It iterates > all mutations in order to get the data size. The cost is directly > proportional to the size of batch. We should remove it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17778) Remove the testing code in the AsyncRequestFutureImpl
[ https://issues.apache.org/jira/browse/HBASE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17778: --- Attachment: HBASE-17723.v1.patch add trivial change to test hbase-server > Remove the testing code in the AsyncRequestFutureImpl > - > > Key: HBASE-17778 > URL: https://issues.apache.org/jira/browse/HBASE-17778 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17778.v0.patch > > > HBASE-16224 left some testing code in the AsyncRequestFutureImpl. It iterates > all mutations in order to get the data size. The cost is directly > proportional to the size of batch. We should remove it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17778) Remove the testing code in the AsyncRequestFutureImpl
[ https://issues.apache.org/jira/browse/HBASE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17778: --- Attachment: (was: HBASE-17723.v1.patch) > Remove the testing code in the AsyncRequestFutureImpl > - > > Key: HBASE-17778 > URL: https://issues.apache.org/jira/browse/HBASE-17778 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17778.v0.patch > > > HBASE-16224 left some testing code in the AsyncRequestFutureImpl. It iterates > all mutations in order to get the data size. The cost is directly > proportional to the size of batch. We should remove it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17778) Remove the testing code in the AsyncRequestFutureImpl
[ https://issues.apache.org/jira/browse/HBASE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17778: --- Attachment: HBASE-17778.v1.patch > Remove the testing code in the AsyncRequestFutureImpl > - > > Key: HBASE-17778 > URL: https://issues.apache.org/jira/browse/HBASE-17778 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17778.v0.patch, HBASE-17778.v1.patch > > > HBASE-16224 left some testing code in the AsyncRequestFutureImpl. It iterates > all mutations in order to get the data size. The cost is directly > proportional to the size of batch. We should remove it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17778) Remove the testing code in the AsyncRequestFutureImpl
[ https://issues.apache.org/jira/browse/HBASE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17778: --- Status: Patch Available (was: Open) > Remove the testing code in the AsyncRequestFutureImpl > - > > Key: HBASE-17778 > URL: https://issues.apache.org/jira/browse/HBASE-17778 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17778.v0.patch, HBASE-17778.v1.patch > > > HBASE-16224 left some testing code in the AsyncRequestFutureImpl. It iterates > all mutations in order to get the data size. The cost is directly > proportional to the size of batch. We should remove it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17453) add Ping into HBase server for deprecated GetProtocolVersion
[
https://issues.apache.org/jira/browse/HBASE-17453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907016#comment-15907016
]
Hadoop QA commented on HBASE-17453:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
1s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 37s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 56s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 54s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 17s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 27s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 10s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 96m 25s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
53s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 164m 21s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12857476/HBASE-17453-master.patch
|
| JIRA Issue | HBASE-17453 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile cc hbaseprotoc |
| uname | Linux a37eca734e9e 3.13.0-107-generic #154-Ubuntu SM
[jira] [Commented] (HBASE-17765) Reviving the merge possibility in the CompactingMemStore
[ https://issues.apache.org/jira/browse/HBASE-17765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907030#comment-15907030 ] Edward Bortnikov commented on HBASE-17765: -- Merge means that only the index data is restructured. We create a larger segment with one index - but no data is copied. Also, we avoid using the SQM scan (more expensive), so duplicate data versions are not eliminated. Bottom line - (1) the overhead and the space savings are both between BASIC and EAGER, and (2) the tail read latency problem is solved. We'll be publishing the perf results shortly. Following that, let's collectively decide whether MERGE should be a level between BASIC and EAGER, or maybe just become the new BASIC, for simplicity. Thanks. > Reviving the merge possibility in the CompactingMemStore > > > Key: HBASE-17765 > URL: https://issues.apache.org/jira/browse/HBASE-17765 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: HBASE-17765-V01.patch > > > According to the new performance results presented in the HBASE-16417 we see > that the read latency of the 90th percentile of the BASIC policy is too big > due to the need to traverse through too many segments in the pipeline. In > this JIRA we correct the bug in the merge sizing calculations and allow > pipeline size threshold to be a configurable parameter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16584) Backport the new ipc implementation in HBASE-16432 to branch-1
[
https://issues.apache.org/jira/browse/HBASE-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907050#comment-15907050
]
Yu Li commented on HBASE-16584:
---
Checking the UT result, it seems to me the only failure is irrelative to
changes here:
{noformat}
Failed tests:
TestCompactionInDeadRegionServer.test:132 Should fail as our wal file has
already been closed, and walDir has also been renamed
{noformat}
So I guess this is a good base for review? Mind update the patch in RB sir
[~Apache9]? Thanks.
> Backport the new ipc implementation in HBASE-16432 to branch-1
> --
>
> Key: HBASE-16584
> URL: https://issues.apache.org/jira/browse/HBASE-16584
> Project: HBase
> Issue Type: Task
> Components: Client, IPC/RPC
>Affects Versions: 1.4.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 1.4.0
>
> Attachments: HBASE-16584-branch-1.patch,
> HBASE-16584-branch-1-v1.patch, HBASE-16584-branch-1-v2.patch,
> HBASE-16584-branch-1-v3.patch
>
>
> Two problems.
> First, as RpcCliemtImpl is the default implementation on branch-1, we need to
> confirm that the modification on master does not make it slower. I'm not sure
> but I used a big lock to protect everything in the new implementation, so it
> may have bad impact on performance.
> Second, some configurations of the old AsyncRpcClient is gone. Such as
> "hbase.rpc.client.threads.max" and "hbase.rpc.client.nativetransport". Now
> You could pass a EventLoopGroup object directly through a helper class which
> makes it more flexible.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17778) Remove the testing code in the AsyncRequestFutureImpl
[ https://issues.apache.org/jira/browse/HBASE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907055#comment-15907055 ] Yu Li commented on HBASE-17778: --- LGTM, +1. [~Apache9] could you take a look here? Thanks. > Remove the testing code in the AsyncRequestFutureImpl > - > > Key: HBASE-17778 > URL: https://issues.apache.org/jira/browse/HBASE-17778 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17778.v0.patch, HBASE-17778.v1.patch > > > HBASE-16224 left some testing code in the AsyncRequestFutureImpl. It iterates > all mutations in order to get the data size. The cost is directly > proportional to the size of batch. We should remove it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907057#comment-15907057 ] Ted Yu commented on HBASE-16179: The comment about source tarball has been addressed. See changes to src.xml I want to get Sean's opinion on building two combinations instead of 4. We can add profiles for the two combinations not built by default so that users can build them manually. > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v10.txt, 16179.v11.txt, > 16179.v12.txt, 16179.v12.txt, 16179.v12.txt, 16179.v13.txt, 16179.v15.txt, > 16179.v16.txt, 16179.v18.txt, 16179.v19.txt, 16179.v19.txt, 16179.v1.txt, > 16179.v1.txt, 16179.v20.txt, 16179.v22.txt, 16179.v23.txt, 16179.v4.txt, > 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17765) Reviving the merge possibility in the CompactingMemStore
[ https://issues.apache.org/jira/browse/HBASE-17765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907061#comment-15907061 ] Anastasia Braginsky commented on HBASE-17765: - [~anoop.hbase], in addition to what [~ebortnik] had answered you, your math is correct. 16 should be max and 30 stands there for safety. According to our recent experiments, we are not sure the default way should be no merge. But let discuss it when we publish the results. Currently the #segments in pipeline is configurable and can be any number. Pay attention that this patch also fixes the bug according to which the region server wasn't updated with the new sizes when the merge did happened. > Reviving the merge possibility in the CompactingMemStore > > > Key: HBASE-17765 > URL: https://issues.apache.org/jira/browse/HBASE-17765 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: HBASE-17765-V01.patch > > > According to the new performance results presented in the HBASE-16417 we see > that the read latency of the 90th percentile of the BASIC policy is too big > due to the need to traverse through too many segments in the pipeline. In > this JIRA we correct the bug in the merge sizing calculations and allow > pipeline size threshold to be a configurable parameter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17765) Reviving the merge possibility in the CompactingMemStore
[ https://issues.apache.org/jira/browse/HBASE-17765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907066#comment-15907066 ] ramkrishna.s.vasudevan commented on HBASE-17765: REading thro the patch, now since you have increased the threshold to 30, you will not do the MERGE till 30 is reached and so the data will be read from multiple segments only. But you will keep flattening the youngest segment? > Reviving the merge possibility in the CompactingMemStore > > > Key: HBASE-17765 > URL: https://issues.apache.org/jira/browse/HBASE-17765 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: HBASE-17765-V01.patch > > > According to the new performance results presented in the HBASE-16417 we see > that the read latency of the 90th percentile of the BASIC policy is too big > due to the need to traverse through too many segments in the pipeline. In > this JIRA we correct the bug in the merge sizing calculations and allow > pipeline size threshold to be a configurable parameter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17758) [RSGROUP] Add shell command to move servers and tables at the same time
[
https://issues.apache.org/jira/browse/HBASE-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907072#comment-15907072
]
Ted Yu commented on HBASE-17758:
In the test, I see two failure scenarios:
{code}
+} catch(IOException ex) {
...
+ assertTrue(msg + " " + ex.getMessage(), ex.getMessage().contains(exp));
{code}
They are for illegal movements.
Where is the test w.r.t. consistency maintenance. I thought some custom
observer should be involved to produce server movement failure.
> [RSGROUP] Add shell command to move servers and tables at the same time
> ---
>
> Key: HBASE-17758
> URL: https://issues.apache.org/jira/browse/HBASE-17758
> Project: HBase
> Issue Type: New Feature
> Components: rsgroup
>Affects Versions: 2.0.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
> Attachments: HBASE-17758-v1.patch, HBASE-17758-v2.patch,
> HBASE-17758-v3.patch, HBASE-17758-v4.patch
>
>
> Currently add a new group perform the following steps:
> {code:javascript}
> hbase(main):001:0> add_rsgroup 'mygroup'
> Took 0.3840 seconds
> hbase(main):002:0> move_servers_rsgroup
> 'mygroup',['hbase-rs-01:16030','hbase-rs-02:16030']
> Took 3.5040 seconds
> hbase(main):003:0> move_tables_rsgroup 'mygroup',['example']
> Took 0.2710 seconds
> {code}
> 1. move_servers_rsgroup will unassign all regions on hbase-rs-01 and
> hbase-rs-02
> 2. move_tables_rsgroup will also reassign all the regions of the table
> example.
> This will lead to a large number of regions to migrate and affect the data
> locality.
> However, some regions of the table that are distributed on hbase-rs-01 and
> hbase-rs-02, do not need to be moved.
> So,we need a new shell command *move_servers_tables_rsgroup* which minimizes
> the number of regions needed to move.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-15143) Procedure v2 - Web UI displaying queues
[
https://issues.apache.org/jira/browse/HBASE-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907074#comment-15907074
]
Balazs Meszaros commented on HBASE-15143:
-
Lock listing from shell:
{code}
hbase(main):001:0> list_locks
NAMESPACE(ns4)
Lock type: SHARED, count: 1
TABLE(ns4:table4)
Lock type: EXCLUSIVE, procedure: 1
Waiting procedures:
Lock type Procedure Id
SHARED 2
EXCLUSIVE 3
2 row(s)
{code}
Lock listing from web:
(see screenshot.png)
> Procedure v2 - Web UI displaying queues
> ---
>
> Key: HBASE-15143
> URL: https://issues.apache.org/jira/browse/HBASE-15143
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2, UI
>Reporter: Matteo Bertozzi
>Assignee: Balazs Meszaros
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-15143-BM-0001.patch, HBASE-15143-BM-0002.patch
>
>
> We can query MasterProcedureScheduler to display the various procedures and
> who is holding table/region locks.
> Each procedure is in a TableQueue or ServerQueue, so it is easy to display
> the procedures in its own group.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-15143) Procedure v2 - Web UI displaying queues
[ https://issues.apache.org/jira/browse/HBASE-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balazs Meszaros updated HBASE-15143: Attachment: screenshot.png > Procedure v2 - Web UI displaying queues > --- > > Key: HBASE-15143 > URL: https://issues.apache.org/jira/browse/HBASE-15143 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, UI >Reporter: Matteo Bertozzi >Assignee: Balazs Meszaros >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15143-BM-0001.patch, HBASE-15143-BM-0002.patch > > > We can query MasterProcedureScheduler to display the various procedures and > who is holding table/region locks. > Each procedure is in a TableQueue or ServerQueue, so it is easy to display > the procedures in its own group. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15143) Procedure v2 - Web UI displaying queues
[ https://issues.apache.org/jira/browse/HBASE-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balazs Meszaros updated HBASE-15143: Attachment: (was: screenshot.png) > Procedure v2 - Web UI displaying queues > --- > > Key: HBASE-15143 > URL: https://issues.apache.org/jira/browse/HBASE-15143 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, UI >Reporter: Matteo Bertozzi >Assignee: Balazs Meszaros >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15143-BM-0001.patch, HBASE-15143-BM-0002.patch > > > We can query MasterProcedureScheduler to display the various procedures and > who is holding table/region locks. > Each procedure is in a TableQueue or ServerQueue, so it is easy to display > the procedures in its own group. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15143) Procedure v2 - Web UI displaying queues
[ https://issues.apache.org/jira/browse/HBASE-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balazs Meszaros updated HBASE-15143: Attachment: screenshot.png > Procedure v2 - Web UI displaying queues > --- > > Key: HBASE-15143 > URL: https://issues.apache.org/jira/browse/HBASE-15143 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, UI >Reporter: Matteo Bertozzi >Assignee: Balazs Meszaros >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15143-BM-0001.patch, HBASE-15143-BM-0002.patch, > screenshot.png > > > We can query MasterProcedureScheduler to display the various procedures and > who is holding table/region locks. > Each procedure is in a TableQueue or ServerQueue, so it is easy to display > the procedures in its own group. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15143) Procedure v2 - Web UI displaying queues
[ https://issues.apache.org/jira/browse/HBASE-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balazs Meszaros updated HBASE-15143: Attachment: HBASE-15143-BM-0003.patch > Procedure v2 - Web UI displaying queues > --- > > Key: HBASE-15143 > URL: https://issues.apache.org/jira/browse/HBASE-15143 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, UI >Reporter: Matteo Bertozzi >Assignee: Balazs Meszaros >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15143-BM-0001.patch, HBASE-15143-BM-0002.patch, > HBASE-15143-BM-0003.patch, screenshot.png > > > We can query MasterProcedureScheduler to display the various procedures and > who is holding table/region locks. > Each procedure is in a TableQueue or ServerQueue, so it is easy to display > the procedures in its own group. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15143) Procedure v2 - Web UI displaying queues
[ https://issues.apache.org/jira/browse/HBASE-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balazs Meszaros updated HBASE-15143: Status: Open (was: Patch Available) > Procedure v2 - Web UI displaying queues > --- > > Key: HBASE-15143 > URL: https://issues.apache.org/jira/browse/HBASE-15143 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, UI >Reporter: Matteo Bertozzi >Assignee: Balazs Meszaros >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15143-BM-0001.patch, HBASE-15143-BM-0002.patch, > HBASE-15143-BM-0003.patch, screenshot.png > > > We can query MasterProcedureScheduler to display the various procedures and > who is holding table/region locks. > Each procedure is in a TableQueue or ServerQueue, so it is easy to display > the procedures in its own group. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15143) Procedure v2 - Web UI displaying queues
[ https://issues.apache.org/jira/browse/HBASE-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balazs Meszaros updated HBASE-15143: Fix Version/s: (was: 1.4.0) Status: Patch Available (was: Open) > Procedure v2 - Web UI displaying queues > --- > > Key: HBASE-15143 > URL: https://issues.apache.org/jira/browse/HBASE-15143 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, UI >Reporter: Matteo Bertozzi >Assignee: Balazs Meszaros >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15143-BM-0001.patch, HBASE-15143-BM-0002.patch, > HBASE-15143-BM-0003.patch, screenshot.png > > > We can query MasterProcedureScheduler to display the various procedures and > who is holding table/region locks. > Each procedure is in a TableQueue or ServerQueue, so it is easy to display > the procedures in its own group. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17773) VerifyReplication tool wrongly emits warning "ERROR: Invalid argument '--recomparesleep=xx'"
[
https://issues.apache.org/jira/browse/HBASE-17773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907088#comment-15907088
]
Guanghao Zhang commented on HBASE-17773:
+1.
> VerifyReplication tool wrongly emits warning "ERROR: Invalid argument
> '--recomparesleep=xx'"
>
>
> Key: HBASE-17773
> URL: https://issues.apache.org/jira/browse/HBASE-17773
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Tomu Tsuruhara
>Assignee: Tomu Tsuruhara
>Priority: Trivial
> Attachments: HBASE-17773.master.001.patch
>
>
> Even though it's completely valid, VerifyReplication tool says "Invalid
> argument" when specifying {{\-\-recomparesleep}} or {{\-\-delimiter}} option.
> {noformat}
> $ bin/hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication
> --recomparesleep=5 1 foo
> ERROR: Invalid argument '--recomparesleep=5'
> Usage: verifyrep [--starttime=X] [--endtime=Y] [--families=A]
> [--row-prefixes=B] [--delimiter=] [--recomparesleep=] [--verbose]
>
> Options:
> starttimebeginning of the time range
> without endtime means from starttime to forever
> endtime end of the time range
> versions number of cell versions to verify
> raw includes raw scan if given in options
> families comma-separated list of families to copy
> row-prefixes comma-separated list of row key prefixes to filter on
> delimiterthe delimiter used in display around rowkey
> recomparesleep milliseconds to sleep before recompare row, default value
> is 0 which disables the recompare.
> verbose logs row keys of good rows
> Args:
> peerid Id of the peer used for verification, must match the one given
> for replication
> tablenameName of the table to verify
> Examples:
> To verify the data replicated from TestTable for a 1 hour window with peer #5
> $ hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication
> --starttime=1265875194289 --endtime=1265878794289 5 TestTable
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17773) VerifyReplication tool wrongly emits warning "ERROR: Invalid argument '--recomparesleep=xx'"
[
https://issues.apache.org/jira/browse/HBASE-17773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-17773:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Pushed to master and branch-1. Thanks [~tomu.tsuruhara] for the patch.
> VerifyReplication tool wrongly emits warning "ERROR: Invalid argument
> '--recomparesleep=xx'"
>
>
> Key: HBASE-17773
> URL: https://issues.apache.org/jira/browse/HBASE-17773
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Tomu Tsuruhara
>Assignee: Tomu Tsuruhara
>Priority: Trivial
> Attachments: HBASE-17773.master.001.patch
>
>
> Even though it's completely valid, VerifyReplication tool says "Invalid
> argument" when specifying {{\-\-recomparesleep}} or {{\-\-delimiter}} option.
> {noformat}
> $ bin/hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication
> --recomparesleep=5 1 foo
> ERROR: Invalid argument '--recomparesleep=5'
> Usage: verifyrep [--starttime=X] [--endtime=Y] [--families=A]
> [--row-prefixes=B] [--delimiter=] [--recomparesleep=] [--verbose]
>
> Options:
> starttimebeginning of the time range
> without endtime means from starttime to forever
> endtime end of the time range
> versions number of cell versions to verify
> raw includes raw scan if given in options
> families comma-separated list of families to copy
> row-prefixes comma-separated list of row key prefixes to filter on
> delimiterthe delimiter used in display around rowkey
> recomparesleep milliseconds to sleep before recompare row, default value
> is 0 which disables the recompare.
> verbose logs row keys of good rows
> Args:
> peerid Id of the peer used for verification, must match the one given
> for replication
> tablenameName of the table to verify
> Examples:
> To verify the data replicated from TestTable for a 1 hour window with peer #5
> $ hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication
> --starttime=1265875194289 --endtime=1265878794289 5 TestTable
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17773) VerifyReplication tool wrongly emits warning "ERROR: Invalid argument '--recomparesleep=xx'"
[
https://issues.apache.org/jira/browse/HBASE-17773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-17773:
---
Fix Version/s: 1.4.0
2.0.0
> VerifyReplication tool wrongly emits warning "ERROR: Invalid argument
> '--recomparesleep=xx'"
>
>
> Key: HBASE-17773
> URL: https://issues.apache.org/jira/browse/HBASE-17773
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Tomu Tsuruhara
>Assignee: Tomu Tsuruhara
>Priority: Trivial
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17773.master.001.patch
>
>
> Even though it's completely valid, VerifyReplication tool says "Invalid
> argument" when specifying {{\-\-recomparesleep}} or {{\-\-delimiter}} option.
> {noformat}
> $ bin/hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication
> --recomparesleep=5 1 foo
> ERROR: Invalid argument '--recomparesleep=5'
> Usage: verifyrep [--starttime=X] [--endtime=Y] [--families=A]
> [--row-prefixes=B] [--delimiter=] [--recomparesleep=] [--verbose]
>
> Options:
> starttimebeginning of the time range
> without endtime means from starttime to forever
> endtime end of the time range
> versions number of cell versions to verify
> raw includes raw scan if given in options
> families comma-separated list of families to copy
> row-prefixes comma-separated list of row key prefixes to filter on
> delimiterthe delimiter used in display around rowkey
> recomparesleep milliseconds to sleep before recompare row, default value
> is 0 which disables the recompare.
> verbose logs row keys of good rows
> Args:
> peerid Id of the peer used for verification, must match the one given
> for replication
> tablenameName of the table to verify
> Examples:
> To verify the data replicated from TestTable for a 1 hour window with peer #5
> $ hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication
> --starttime=1265875194289 --endtime=1265878794289 5 TestTable
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-15143) Procedure v2 - Web UI displaying queues
[ https://issues.apache.org/jira/browse/HBASE-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balazs Meszaros updated HBASE-15143: Status: Open (was: Patch Available) > Procedure v2 - Web UI displaying queues > --- > > Key: HBASE-15143 > URL: https://issues.apache.org/jira/browse/HBASE-15143 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, UI >Reporter: Matteo Bertozzi >Assignee: Balazs Meszaros >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15143-BM-0001.patch, HBASE-15143-BM-0002.patch, > HBASE-15143-BM-0003.patch, screenshot.png > > > We can query MasterProcedureScheduler to display the various procedures and > who is holding table/region locks. > Each procedure is in a TableQueue or ServerQueue, so it is easy to display > the procedures in its own group. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16584) Backport the new ipc implementation in HBASE-16432 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907110#comment-15907110 ] Duo Zhang commented on HBASE-16584: --- I've uploaded it several days ago... See the link above, and it is also in comments... > Backport the new ipc implementation in HBASE-16432 to branch-1 > -- > > Key: HBASE-16584 > URL: https://issues.apache.org/jira/browse/HBASE-16584 > Project: HBase > Issue Type: Task > Components: Client, IPC/RPC >Affects Versions: 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 1.4.0 > > Attachments: HBASE-16584-branch-1.patch, > HBASE-16584-branch-1-v1.patch, HBASE-16584-branch-1-v2.patch, > HBASE-16584-branch-1-v3.patch > > > Two problems. > First, as RpcCliemtImpl is the default implementation on branch-1, we need to > confirm that the modification on master does not make it slower. I'm not sure > but I used a big lock to protect everything in the new implementation, so it > may have bad impact on performance. > Second, some configurations of the old AsyncRpcClient is gone. Such as > "hbase.rpc.client.threads.max" and "hbase.rpc.client.nativetransport". Now > You could pass a EventLoopGroup object directly through a helper class which > makes it more flexible. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17778) Remove the testing code in the AsyncRequestFutureImpl
[ https://issues.apache.org/jira/browse/HBASE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907115#comment-15907115 ] Duo Zhang commented on HBASE-17778: --- Skimmed the patch, overall LGTM. +1 if it could pass all the UTs. > Remove the testing code in the AsyncRequestFutureImpl > - > > Key: HBASE-17778 > URL: https://issues.apache.org/jira/browse/HBASE-17778 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17778.v0.patch, HBASE-17778.v1.patch > > > HBASE-16224 left some testing code in the AsyncRequestFutureImpl. It iterates > all mutations in order to get the data size. The cost is directly > proportional to the size of batch. We should remove it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17747) Support both weak and soft object pool
[
https://issues.apache.org/jira/browse/HBASE-17747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907134#comment-15907134
]
Yu Li commented on HBASE-17747:
---
Ok, here comes the E2E test report through YCSB (with 100% cache hit ratio):
|| Case || Off-heap || Round || Throughput || AverageThroughput||
| w/o patch | No | 1 | 275652.627079| |
| | | 2 | 248664.37| |
| | | 3 | 264297.580882| |
| | | 4 | 253694.307118| |
| | | | | 260577.212104 |
| w/ patch | No | 1 | 268122.883638| |
| | | 2 | 284483.969107| |
| | | 3 | 283008.465412| |
| | | 4 | 268476.579267| |
| | | | | 276022.974356 (+5.9%) |
| w/o patch | Yes | 1 | 199073.764764| |
| | | 2 | 210060.769909| |
| | | 3 | 216729.412051| |
| | | 4 | 221610.155932| |
| | | | | 211868.525664 |
| w/ patch | Yes | 1 | 217398.730766| |
| | | 2 | 232581.157272| |
| | | 3 | 225765.026296| |
| | | 4 | 229071.140011| |
| | | | | 226204.013586 (+6.8%) |
>From the result we could see with soft reference we get a better performance.
More details about testing configuration:
{noformat}
YCSB:
* 4 physical nodes, 8 ycsb processes on each node
* recordcount=1100
* fieldcount=1
* fieldlength=1024 (10.8GB overall)
* threadcount=8
* maxexecutiontime=300
* table schema: {NAME => 'cf', DATA_BLOCK_ENCODING => 'DIFF', VERSIONS=> '1',
COMPRESSION => 'SNAPPY', IN_MEMORY => 'false', BLOCKCACHE => 'true'},{SPLITS =>
(1..9).map {|i| "user#{1000+i*(-1000)/9}"},
MAX_FILESIZE=>'223372036854775807',DURABILITY=>'SYNC_WAL',METADATA =>
{'hbase.hstore.block.storage.policy' => 'ALL_SSD'}}
HBase:
* One single regionserver, 3 datanode
* base commit SHA (w/o patch): 31bc94ae60
* hbase.regionserver.handler.count=192
* On-heap: -Xmx42g, hfile.block.cache.size=0.3
* Off-heap: hbase.bucketcache.size=12288,
hbase.bucketcache.writer.queuelength=64, hbase.bucketcache.writer.threads=3
{noformat}
> Support both weak and soft object pool
> --
>
> Key: HBASE-17747
> URL: https://issues.apache.org/jira/browse/HBASE-17747
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17747.patch, HBASE-17747.v2.patch,
> HBASE-17747.v3.patch
>
>
> During YCSB testing on embedded mode after HBASE-17744, we found that under
> high read load GC is quite severe even with offheap L2 cache. After some
> investigation, we found it's caused by using weak reference for
> IdReadWriteLock. In embedded mode the read is so quick that the lock might
> already get promoted to the old generation when the weak reference is
> cleared, which causes dirty card table thus slowing YGC.
> So we proposed to use soft reference for this IdReadWriteLock used in cache,
> which won't get cleared until JVM memory is not enough, and could resolve the
> issue mentioned above. What's more, we propose to extend the WeakObjectPool
> to be more generate to support both weak and soft reference.
> Note that this issue only emerges under embedded mode with DirectOperator, in
> which case all costs on the wire is removed thus produces extremely high
> workloads.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17747) Support both weak and soft object pool
[
https://issues.apache.org/jira/browse/HBASE-17747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-17747:
--
Description:
During YCSB testing on embedded mode after HBASE-17744, we found that under
high read load GC is quite severe even with offheap L2 cache. After some
investigation, we found it's caused by using weak reference in
{{IdReadWriteLock}}. In embedded mode the read is so quick that the lock might
already get promoted to the old generation when the weak reference is cleared,
which causes dirty card table (old reference get removed and new lock object
set into {{referenceCache}}, see {{WeakObjectPool#get}}) thus slowing YGC. In
distributed mode there'll also be more lock object created with weak reference
than soft reference that slowing down the processing.
So we proposed to use soft reference for this {{IdReadWriteLock}} used in
cache, which won't get cleared until JVM memory is not enough, and could
resolve the issue mentioned above. What's more, we propose to extend the
{{WeakObjectPool}} to be more generate to support both weak and soft reference.
Note that the GC issue only emerges under embedded mode with DirectOperator, in
which case all costs on the wire is removed thus produces extremely high
concurrency.
was:
During YCSB testing on embedded mode after HBASE-17744, we found that under
high read load GC is quite severe even with offheap L2 cache. After some
investigation, we found it's caused by using weak reference for
IdReadWriteLock. In embedded mode the read is so quick that the lock might
already get promoted to the old generation when the weak reference is cleared,
which causes dirty card table thus slowing YGC.
So we proposed to use soft reference for this IdReadWriteLock used in cache,
which won't get cleared until JVM memory is not enough, and could resolve the
issue mentioned above. What's more, we propose to extend the WeakObjectPool to
be more generate to support both weak and soft reference.
Note that this issue only emerges under embedded mode with DirectOperator, in
which case all costs on the wire is removed thus produces extremely high
workloads.
> Support both weak and soft object pool
> --
>
> Key: HBASE-17747
> URL: https://issues.apache.org/jira/browse/HBASE-17747
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17747.patch, HBASE-17747.v2.patch,
> HBASE-17747.v3.patch
>
>
> During YCSB testing on embedded mode after HBASE-17744, we found that under
> high read load GC is quite severe even with offheap L2 cache. After some
> investigation, we found it's caused by using weak reference in
> {{IdReadWriteLock}}. In embedded mode the read is so quick that the lock
> might already get promoted to the old generation when the weak reference is
> cleared, which causes dirty card table (old reference get removed and new
> lock object set into {{referenceCache}}, see {{WeakObjectPool#get}}) thus
> slowing YGC. In distributed mode there'll also be more lock object created
> with weak reference than soft reference that slowing down the processing.
> So we proposed to use soft reference for this {{IdReadWriteLock}} used in
> cache, which won't get cleared until JVM memory is not enough, and could
> resolve the issue mentioned above. What's more, we propose to extend the
> {{WeakObjectPool}} to be more generate to support both weak and soft
> reference.
> Note that the GC issue only emerges under embedded mode with DirectOperator,
> in which case all costs on the wire is removed thus produces extremely high
> concurrency.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17747) Support both weak and soft object pool
[
https://issues.apache.org/jira/browse/HBASE-17747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907148#comment-15907148
]
Yu Li commented on HBASE-17747:
---
[~anoop.hbase] Does the performance result look good to you sir? Thanks.
Plan to commit this in a day or so if no objections.
> Support both weak and soft object pool
> --
>
> Key: HBASE-17747
> URL: https://issues.apache.org/jira/browse/HBASE-17747
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17747.patch, HBASE-17747.v2.patch,
> HBASE-17747.v3.patch
>
>
> During YCSB testing on embedded mode after HBASE-17744, we found that under
> high read load GC is quite severe even with offheap L2 cache. After some
> investigation, we found it's caused by using weak reference in
> {{IdReadWriteLock}}. In embedded mode the read is so quick that the lock
> might already get promoted to the old generation when the weak reference is
> cleared, which causes dirty card table (old reference get removed and new
> lock object set into {{referenceCache}}, see {{WeakObjectPool#get}}) thus
> slowing YGC. In distributed mode there'll also be more lock object created
> with weak reference than soft reference that slowing down the processing.
> So we proposed to use soft reference for this {{IdReadWriteLock}} used in
> cache, which won't get cleared until JVM memory is not enough, and could
> resolve the issue mentioned above. What's more, we propose to extend the
> {{WeakObjectPool}} to be more generate to support both weak and soft
> reference.
> Note that the GC issue only emerges under embedded mode with DirectOperator,
> in which case all costs on the wire is removed thus produces extremely high
> concurrency.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17747) Support both weak and soft object pool
[
https://issues.apache.org/jira/browse/HBASE-17747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907152#comment-15907152
]
Yu Li commented on HBASE-17747:
---
Linking to HBASE-14463 which originally introduced {{IdReadWriteLock}}.
> Support both weak and soft object pool
> --
>
> Key: HBASE-17747
> URL: https://issues.apache.org/jira/browse/HBASE-17747
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17747.patch, HBASE-17747.v2.patch,
> HBASE-17747.v3.patch
>
>
> During YCSB testing on embedded mode after HBASE-17744, we found that under
> high read load GC is quite severe even with offheap L2 cache. After some
> investigation, we found it's caused by using weak reference in
> {{IdReadWriteLock}}. In embedded mode the read is so quick that the lock
> might already get promoted to the old generation when the weak reference is
> cleared, which causes dirty card table (old reference get removed and new
> lock object set into {{referenceCache}}, see {{WeakObjectPool#get}}) thus
> slowing YGC. In distributed mode there'll also be more lock object created
> with weak reference than soft reference that slowing down the processing.
> So we proposed to use soft reference for this {{IdReadWriteLock}} used in
> cache, which won't get cleared until JVM memory is not enough, and could
> resolve the issue mentioned above. What's more, we propose to extend the
> {{WeakObjectPool}} to be more generate to support both weak and soft
> reference.
> Note that the GC issue only emerges under embedded mode with DirectOperator,
> in which case all costs on the wire is removed thus produces extremely high
> concurrency.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-16584) Backport the new ipc implementation in HBASE-16432 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907154#comment-15907154 ] Yu Li commented on HBASE-16584: --- So it's already v3 patch in rb? Just to confirm... Thanks. > Backport the new ipc implementation in HBASE-16432 to branch-1 > -- > > Key: HBASE-16584 > URL: https://issues.apache.org/jira/browse/HBASE-16584 > Project: HBase > Issue Type: Task > Components: Client, IPC/RPC >Affects Versions: 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 1.4.0 > > Attachments: HBASE-16584-branch-1.patch, > HBASE-16584-branch-1-v1.patch, HBASE-16584-branch-1-v2.patch, > HBASE-16584-branch-1-v3.patch > > > Two problems. > First, as RpcCliemtImpl is the default implementation on branch-1, we need to > confirm that the modification on master does not make it slower. I'm not sure > but I used a big lock to protect everything in the new implementation, so it > may have bad impact on performance. > Second, some configurations of the old AsyncRpcClient is gone. Such as > "hbase.rpc.client.threads.max" and "hbase.rpc.client.nativetransport". Now > You could pass a EventLoopGroup object directly through a helper class which > makes it more flexible. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16584) Backport the new ipc implementation in HBASE-16432 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907158#comment-15907158 ] Duo Zhang commented on HBASE-16584: --- Yes. > Backport the new ipc implementation in HBASE-16432 to branch-1 > -- > > Key: HBASE-16584 > URL: https://issues.apache.org/jira/browse/HBASE-16584 > Project: HBase > Issue Type: Task > Components: Client, IPC/RPC >Affects Versions: 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 1.4.0 > > Attachments: HBASE-16584-branch-1.patch, > HBASE-16584-branch-1-v1.patch, HBASE-16584-branch-1-v2.patch, > HBASE-16584-branch-1-v3.patch > > > Two problems. > First, as RpcCliemtImpl is the default implementation on branch-1, we need to > confirm that the modification on master does not make it slower. I'm not sure > but I used a big lock to protect everything in the new implementation, so it > may have bad impact on performance. > Second, some configurations of the old AsyncRpcClient is gone. Such as > "hbase.rpc.client.threads.max" and "hbase.rpc.client.nativetransport". Now > You could pass a EventLoopGroup object directly through a helper class which > makes it more flexible. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17758) [RSGROUP] Add shell command to move servers and tables at the same time
[
https://issues.apache.org/jira/browse/HBASE-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907166#comment-15907166
]
Hadoop QA commented on HBASE-17758:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 0s
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 0s
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 20 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 1s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 100m 37s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 24s {color}
| {color:red} hbase-rsgroup in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 58s
{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 159m 11s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.rsgroup.TestRSGroups |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12857574/HBASE-17758-v4.patch |
| JIRA Issue | HBASE-17758 |
| Optional Tests | asflicense javac javado
[jira] [Updated] (HBASE-17778) Remove the testing code in the AsyncRequestFutureImpl
[ https://issues.apache.org/jira/browse/HBASE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17778: --- Fix Version/s: 1.4.0 > Remove the testing code in the AsyncRequestFutureImpl > - > > Key: HBASE-17778 > URL: https://issues.apache.org/jira/browse/HBASE-17778 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0, 1.4.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Trivial > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17778.v0.patch, HBASE-17778.v1.patch > > > HBASE-16224 left some testing code in the AsyncRequestFutureImpl. It iterates > all mutations in order to get the data size. The cost is directly > proportional to the size of batch. We should remove it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17778) Remove the testing code in the AsyncRequestFutureImpl
[ https://issues.apache.org/jira/browse/HBASE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17778: --- Affects Version/s: 1.4.0 > Remove the testing code in the AsyncRequestFutureImpl > - > > Key: HBASE-17778 > URL: https://issues.apache.org/jira/browse/HBASE-17778 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0, 1.4.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Trivial > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17778.v0.patch, HBASE-17778.v1.patch > > > HBASE-16224 left some testing code in the AsyncRequestFutureImpl. It iterates > all mutations in order to get the data size. The cost is directly > proportional to the size of batch. We should remove it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17778) Remove the testing code in the AsyncRequestFutureImpl
[
https://issues.apache.org/jira/browse/HBASE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907290#comment-15907290
]
Hadoop QA commented on HBASE-17778:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 34s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
3s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
32m 59s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 24s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 123m 5s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 179m 9s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12857579/HBASE-17778.v1.patch |
| JIRA Issue | HBASE-17778 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 21943c4d21a4 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 31bc94a |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6075/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6075/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Remove the testing code in the Asy
[jira] [Updated] (HBASE-17778) Remove the testing code in the AsyncRequestFutureImpl
[ https://issues.apache.org/jira/browse/HBASE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17778: --- Status: Open (was: Patch Available) > Remove the testing code in the AsyncRequestFutureImpl > - > > Key: HBASE-17778 > URL: https://issues.apache.org/jira/browse/HBASE-17778 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0, 1.4.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Trivial > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17778.branch-1.v0.patch, HBASE-17778.v0.patch, > HBASE-17778.v1.patch > > > HBASE-16224 left some testing code in the AsyncRequestFutureImpl. It iterates > all mutations in order to get the data size. The cost is directly > proportional to the size of batch. We should remove it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17778) Remove the testing code in the AsyncRequestFutureImpl
[ https://issues.apache.org/jira/browse/HBASE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17778: --- Attachment: HBASE-17778.branch-1.v0.patch > Remove the testing code in the AsyncRequestFutureImpl > - > > Key: HBASE-17778 > URL: https://issues.apache.org/jira/browse/HBASE-17778 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0, 1.4.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Trivial > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17778.branch-1.v0.patch, HBASE-17778.v0.patch, > HBASE-17778.v1.patch > > > HBASE-16224 left some testing code in the AsyncRequestFutureImpl. It iterates > all mutations in order to get the data size. The cost is directly > proportional to the size of batch. We should remove it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17778) Remove the testing code in the AsyncRequestFutureImpl
[ https://issues.apache.org/jira/browse/HBASE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17778: --- Status: Patch Available (was: Open) > Remove the testing code in the AsyncRequestFutureImpl > - > > Key: HBASE-17778 > URL: https://issues.apache.org/jira/browse/HBASE-17778 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0, 1.4.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Trivial > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17778.branch-1.v0.patch, HBASE-17778.v0.patch, > HBASE-17778.v1.patch > > > HBASE-16224 left some testing code in the AsyncRequestFutureImpl. It iterates > all mutations in order to get the data size. The cost is directly > proportional to the size of batch. We should remove it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17460) enable_table_replication can not perform cyclic replication of a table
[ https://issues.apache.org/jira/browse/HBASE-17460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907310#comment-15907310 ] Bhupendra Kumar Jain commented on HBASE-17460: -- bq. Currently HBaseAdmin#isTableRepEnabled() returns false if ANY of the columns families is not replicated, meaning that if you have a table with even one column family without replication you could not disable replication. I think , This is a bug and we should fix it. bq. Admin#disableTableReplication() returns nothing and isTableRepEnabled() returns boolean. For master branch, we can let isTableRepEnabled() return enum (partially disabled, disabled, etc) This way, Admin#disableTableReplication() can return meaningful value to the user. Agree. We can enhance this in master branch > enable_table_replication can not perform cyclic replication of a table > -- > > Key: HBASE-17460 > URL: https://issues.apache.org/jira/browse/HBASE-17460 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: NITIN VERMA >Assignee: NITIN VERMA > Labels: replication > Fix For: 2.0.0, 1.4.0 > > Attachments: 17460-addendum.txt, 17460-addendum.v2.txt, > 17460.branch-1.v3.txt, 17460.v5.txt, HBASE-17460-addendum2.patch, > HBASE-17460-addendum2-v2.patch, HBASE-17460-addendum2-v3.patch, > HBASE-17460-addendum2-v4.patch, > HBASE-17460-branch-1-v5-addendumv2-addendum2v4.patch, > HBASE-17460-branch-1-v5-addendumv2-addendum2v4-v2.patch, > HBASE-17460-branch-1-v5-plusaddendum-v2.patch, HBASE-17460.patch, > HBASE-17460_v2.patch, HBASE-17460_v3.patch, HBASE-17460_v4.patch > > Original Estimate: 96h > Remaining Estimate: 96h > > The enable_table_replication operation is broken for cyclic replication of > HBase table as we compare all the properties of column families (including > REPLICATION_SCOPE). > Below is exactly what happens: > 1. Running "enable_table_replication 'table1' " opeartion on first cluster > will set the REPLICATION_SCOPE of all column families to peer id '1'. This > will also create a table on second cluster where REPLICATION_SCOPE is still > set to peer id '0'. > 2. Now when we run "enable_table_replication 'table1'" on second cluster, we > compare all the properties of table (including REPLICATION_SCOPE_, which > obviously is different now. > I am proposing a fix for this issue where we should avoid comparing > REPLICATION_SCOPE inside HColumnDescriotor::compareTo() method, especially > when replication is not already enabled on the desired table. > I have made that change and it is working. I will submit the patch soon. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17778) Remove the testing code in the AsyncRequestFutureImpl
[ https://issues.apache.org/jira/browse/HBASE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907346#comment-15907346 ] Chia-Ping Tsai commented on HBASE-17778: Thanks for your review [~carp84] [~Apache9]. > Remove the testing code in the AsyncRequestFutureImpl > - > > Key: HBASE-17778 > URL: https://issues.apache.org/jira/browse/HBASE-17778 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0, 1.4.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Trivial > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17778.branch-1.v0.patch, HBASE-17778.v0.patch, > HBASE-17778.v1.patch > > > HBASE-16224 left some testing code in the AsyncRequestFutureImpl. It iterates > all mutations in order to get the data size. The cost is directly > proportional to the size of batch. We should remove it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17778) Remove the testing code in the AsyncRequestFutureImpl
[
https://issues.apache.org/jira/browse/HBASE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907348#comment-15907348
]
Hadoop QA commented on HBASE-17778:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 0s
{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 44s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 48s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 17s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:e01ee2f |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12857661/HBASE-17778.branch-1.v0.patch
|
| JIRA Issue | HBASE-17778 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux d103eff01371 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
| git revision | branch-1 / 759d63b |
| Default Java | 1.7.0_80 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_121
/usr/lib/j
[jira] [Updated] (HBASE-17778) Remove the testing code in the AsyncRequestFutureImpl
[ https://issues.apache.org/jira/browse/HBASE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17778: --- Status: Open (was: Patch Available) > Remove the testing code in the AsyncRequestFutureImpl > - > > Key: HBASE-17778 > URL: https://issues.apache.org/jira/browse/HBASE-17778 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0, 1.4.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Trivial > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17778.branch-1.v0.patch, HBASE-17778.v0.patch, > HBASE-17778.v1.patch > > > HBASE-16224 left some testing code in the AsyncRequestFutureImpl. It iterates > all mutations in order to get the data size. The cost is directly > proportional to the size of batch. We should remove it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17778) Remove the testing code in the AsyncRequestFutureImpl
[ https://issues.apache.org/jira/browse/HBASE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17778: --- Status: Patch Available (was: Open) > Remove the testing code in the AsyncRequestFutureImpl > - > > Key: HBASE-17778 > URL: https://issues.apache.org/jira/browse/HBASE-17778 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0, 1.4.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Trivial > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17778.branch-1.v0.patch, > HBASE-17778.branch-1.v1.patch, HBASE-17778.v0.patch, HBASE-17778.v1.patch > > > HBASE-16224 left some testing code in the AsyncRequestFutureImpl. It iterates > all mutations in order to get the data size. The cost is directly > proportional to the size of batch. We should remove it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17778) Remove the testing code in the AsyncRequestFutureImpl
[ https://issues.apache.org/jira/browse/HBASE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17778: --- Attachment: HBASE-17778.branch-1.v1.patch branch-1.v1 adds trivial change into the hbase-server module. > Remove the testing code in the AsyncRequestFutureImpl > - > > Key: HBASE-17778 > URL: https://issues.apache.org/jira/browse/HBASE-17778 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0, 1.4.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Trivial > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17778.branch-1.v0.patch, > HBASE-17778.branch-1.v1.patch, HBASE-17778.v0.patch, HBASE-17778.v1.patch > > > HBASE-16224 left some testing code in the AsyncRequestFutureImpl. It iterates > all mutations in order to get the data size. The cost is directly > proportional to the size of batch. We should remove it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17779) disable_table_replication returns misleading message and does not turn off repliaton
Janos Gub created HBASE-17779: - Summary: disable_table_replication returns misleading message and does not turn off repliaton Key: HBASE-17779 URL: https://issues.apache.org/jira/browse/HBASE-17779 Project: HBase Issue Type: Bug Reporter: Janos Gub Assignee: Janos Gub Currently HBaseAdmin#isTableRepEnabled() returns false if ANY of the columns families is not replicated. Because of this if you have a table where replication is partially enabled, calling disable_table_replication will not have any effect, but will report that replication for a given table is disabled. Workaround is enabling table replication before disabling. As a solution quoted from HBASE-17460: Admin#disableTableReplication() returns nothing and isTableRepEnabled() returns boolean. For master branch, we can let isTableRepEnabled() return enum (partially disabled, disabled, etc) This way, Admin#disableTableReplication() can return meaningful value to the user. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17460) enable_table_replication can not perform cyclic replication of a table
[ https://issues.apache.org/jira/browse/HBASE-17460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907361#comment-15907361 ] Janos Gub commented on HBASE-17460: --- CreatedHBASE-17779 for the issue in the above comment. > enable_table_replication can not perform cyclic replication of a table > -- > > Key: HBASE-17460 > URL: https://issues.apache.org/jira/browse/HBASE-17460 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: NITIN VERMA >Assignee: NITIN VERMA > Labels: replication > Fix For: 2.0.0, 1.4.0 > > Attachments: 17460-addendum.txt, 17460-addendum.v2.txt, > 17460.branch-1.v3.txt, 17460.v5.txt, HBASE-17460-addendum2.patch, > HBASE-17460-addendum2-v2.patch, HBASE-17460-addendum2-v3.patch, > HBASE-17460-addendum2-v4.patch, > HBASE-17460-branch-1-v5-addendumv2-addendum2v4.patch, > HBASE-17460-branch-1-v5-addendumv2-addendum2v4-v2.patch, > HBASE-17460-branch-1-v5-plusaddendum-v2.patch, HBASE-17460.patch, > HBASE-17460_v2.patch, HBASE-17460_v3.patch, HBASE-17460_v4.patch > > Original Estimate: 96h > Remaining Estimate: 96h > > The enable_table_replication operation is broken for cyclic replication of > HBase table as we compare all the properties of column families (including > REPLICATION_SCOPE). > Below is exactly what happens: > 1. Running "enable_table_replication 'table1' " opeartion on first cluster > will set the REPLICATION_SCOPE of all column families to peer id '1'. This > will also create a table on second cluster where REPLICATION_SCOPE is still > set to peer id '0'. > 2. Now when we run "enable_table_replication 'table1'" on second cluster, we > compare all the properties of table (including REPLICATION_SCOPE_, which > obviously is different now. > I am proposing a fix for this issue where we should avoid comparing > REPLICATION_SCOPE inside HColumnDescriotor::compareTo() method, especially > when replication is not already enabled on the desired table. > I have made that change and it is working. I will submit the patch soon. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17460) enable_table_replication can not perform cyclic replication of a table
[ https://issues.apache.org/jira/browse/HBASE-17460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907361#comment-15907361 ] Janos Gub edited comment on HBASE-17460 at 3/13/17 12:27 PM: - Created HBASE-17779 for the issue in the above comment. was (Author: gubjanos): CreatedHBASE-17779 for the issue in the above comment. > enable_table_replication can not perform cyclic replication of a table > -- > > Key: HBASE-17460 > URL: https://issues.apache.org/jira/browse/HBASE-17460 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: NITIN VERMA >Assignee: NITIN VERMA > Labels: replication > Fix For: 2.0.0, 1.4.0 > > Attachments: 17460-addendum.txt, 17460-addendum.v2.txt, > 17460.branch-1.v3.txt, 17460.v5.txt, HBASE-17460-addendum2.patch, > HBASE-17460-addendum2-v2.patch, HBASE-17460-addendum2-v3.patch, > HBASE-17460-addendum2-v4.patch, > HBASE-17460-branch-1-v5-addendumv2-addendum2v4.patch, > HBASE-17460-branch-1-v5-addendumv2-addendum2v4-v2.patch, > HBASE-17460-branch-1-v5-plusaddendum-v2.patch, HBASE-17460.patch, > HBASE-17460_v2.patch, HBASE-17460_v3.patch, HBASE-17460_v4.patch > > Original Estimate: 96h > Remaining Estimate: 96h > > The enable_table_replication operation is broken for cyclic replication of > HBase table as we compare all the properties of column families (including > REPLICATION_SCOPE). > Below is exactly what happens: > 1. Running "enable_table_replication 'table1' " opeartion on first cluster > will set the REPLICATION_SCOPE of all column families to peer id '1'. This > will also create a table on second cluster where REPLICATION_SCOPE is still > set to peer id '0'. > 2. Now when we run "enable_table_replication 'table1'" on second cluster, we > compare all the properties of table (including REPLICATION_SCOPE_, which > obviously is different now. > I am proposing a fix for this issue where we should avoid comparing > REPLICATION_SCOPE inside HColumnDescriotor::compareTo() method, especially > when replication is not already enabled on the desired table. > I have made that change and it is working. I will submit the patch soon. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15143) Procedure v2 - Web UI displaying queues
[ https://issues.apache.org/jira/browse/HBASE-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balazs Meszaros updated HBASE-15143: Status: Patch Available (was: Open) > Procedure v2 - Web UI displaying queues > --- > > Key: HBASE-15143 > URL: https://issues.apache.org/jira/browse/HBASE-15143 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, UI >Reporter: Matteo Bertozzi >Assignee: Balazs Meszaros >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15143-BM-0001.patch, HBASE-15143-BM-0002.patch, > HBASE-15143-BM-0003.patch, HBASE-15143-BM-0004.patch, screenshot.png > > > We can query MasterProcedureScheduler to display the various procedures and > who is holding table/region locks. > Each procedure is in a TableQueue or ServerQueue, so it is easy to display > the procedures in its own group. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15143) Procedure v2 - Web UI displaying queues
[ https://issues.apache.org/jira/browse/HBASE-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balazs Meszaros updated HBASE-15143: Attachment: HBASE-15143-BM-0004.patch > Procedure v2 - Web UI displaying queues > --- > > Key: HBASE-15143 > URL: https://issues.apache.org/jira/browse/HBASE-15143 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, UI >Reporter: Matteo Bertozzi >Assignee: Balazs Meszaros >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15143-BM-0001.patch, HBASE-15143-BM-0002.patch, > HBASE-15143-BM-0003.patch, HBASE-15143-BM-0004.patch, screenshot.png > > > We can query MasterProcedureScheduler to display the various procedures and > who is holding table/region locks. > Each procedure is in a TableQueue or ServerQueue, so it is easy to display > the procedures in its own group. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17779) disable_table_replication returns misleading message and does not turn off repliaton
[ https://issues.apache.org/jira/browse/HBASE-17779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janos Gub updated HBASE-17779: -- Attachment: HBASE-17779.patch Please find initial patch. In this one enable_table_replication and disable_table_replication always sets the replication scope for a given table, so there is no need to propagate other info to the user. Also it is a behavior change. Should we go for another one reporting when the ReplicationState is mixed? > disable_table_replication returns misleading message and does not turn off > repliaton > > > Key: HBASE-17779 > URL: https://issues.apache.org/jira/browse/HBASE-17779 > Project: HBase > Issue Type: Bug >Reporter: Janos Gub >Assignee: Janos Gub > Attachments: HBASE-17779.patch > > > Currently HBaseAdmin#isTableRepEnabled() returns false if ANY of the columns > families is not replicated. > Because of this if you have a table where replication is partially enabled, > calling disable_table_replication will not have any effect, but will report > that replication for a given table is disabled. > Workaround is enabling table replication before disabling. > As a solution quoted from HBASE-17460: > Admin#disableTableReplication() returns nothing and isTableRepEnabled() > returns boolean. For master branch, we can let isTableRepEnabled() return > enum (partially disabled, disabled, etc) This way, > Admin#disableTableReplication() can return meaningful value to the user. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907440#comment-15907440 ] Anastasia Braginsky commented on HBASE-16438: - Hey [~ram_krish]! Thanks for publishing the new patch! I have looked on it, but didn't publish my detailed comments yet. Here are some of my general thoughts: 1. I agree with [~ram_krish] that we should exclude CellChunkMap implementation from this patch. This patch is big enough and we have here enough design and implementation details. So we should better concentrate on making just this in a great way. My only concern is that if we remove the CellChunkMap we have less abilities to test the new code. We should make sure the new code is well tested. Please add the new tests for all the relevant interfaces (getChunkId() from any cell, translate chunk ID into chunk, etc.). 2. I disagree with the design decision of putting MemStoreLABChunkCreator to be part of MemStoreChunkPool. I also don't understand why MemStoreLABChunkCreator works only with chunks allocated from pool. Please decouple the ChunkPool and the ChunkCreator, there can be chunks without pool, so there should exist ChunkCreator even if it was decided not to allocate/de-allocate Chunks via Pool. Whether it is reasonable or not to have chunks without pool is a separate question and a matter of usage. As we are not sure how exactly CellChunkMap is going to be used in the future we should not block any option for now. As I have already said, the ChunkCreator should be responsible for ANY chunk allocation and ANY chunk should have its ID. Maybe it makes sense to do some small design document? Now I am going to publish more detailed comments on the RB itself. [~ram_krish], thanks once again! > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, > MemstoreChunkCell_trunk.patch > > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17779) disable_table_replication returns misleading message and does not turn off repliaton
[ https://issues.apache.org/jira/browse/HBASE-17779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17779: --- Reporter: Ted Yu (was: Janos Gub) > disable_table_replication returns misleading message and does not turn off > repliaton > > > Key: HBASE-17779 > URL: https://issues.apache.org/jira/browse/HBASE-17779 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Janos Gub > Attachments: HBASE-17779.patch > > > Currently HBaseAdmin#isTableRepEnabled() returns false if ANY of the columns > families is not replicated. > Because of this if you have a table where replication is partially enabled, > calling disable_table_replication will not have any effect, but will report > that replication for a given table is disabled. > Workaround is enabling table replication before disabling. > As a solution quoted from HBASE-17460: > Admin#disableTableReplication() returns nothing and isTableRepEnabled() > returns boolean. For master branch, we can let isTableRepEnabled() return > enum (partially disabled, disabled, etc) This way, > Admin#disableTableReplication() can return meaningful value to the user. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17765) Reviving the merge possibility in the CompactingMemStore
[ https://issues.apache.org/jira/browse/HBASE-17765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907520#comment-15907520 ] Anastasia Braginsky commented on HBASE-17765: - Hi [~ram_krish], Actually I didn't increase the threshold to 30, it was 30 since we turned to use NONE, BASIC and EAGER memstore types. For now for BASIC we actually never merge the segments, but we do flatten the youngest segment. Here we make the threshold configurable because we see it should be actually better to decrease this number. > Reviving the merge possibility in the CompactingMemStore > > > Key: HBASE-17765 > URL: https://issues.apache.org/jira/browse/HBASE-17765 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: HBASE-17765-V01.patch > > > According to the new performance results presented in the HBASE-16417 we see > that the read latency of the 90th percentile of the BASIC policy is too big > due to the need to traverse through too many segments in the pipeline. In > this JIRA we correct the bug in the merge sizing calculations and allow > pipeline size threshold to be a configurable parameter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17773) VerifyReplication tool wrongly emits warning "ERROR: Invalid argument '--recomparesleep=xx'"
[
https://issues.apache.org/jira/browse/HBASE-17773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907558#comment-15907558
]
Hudson commented on HBASE-17773:
FAILURE: Integrated in Jenkins build HBase-1.4 #668 (See
[https://builds.apache.org/job/HBase-1.4/668/])
HBASE-17773 VerifyReplication tool wrongly emits Invalid arguments error
(zghao: rev 759d63b15ca63464dd51d1eeff0f6b7763ab6e01)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.java
> VerifyReplication tool wrongly emits warning "ERROR: Invalid argument
> '--recomparesleep=xx'"
>
>
> Key: HBASE-17773
> URL: https://issues.apache.org/jira/browse/HBASE-17773
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Tomu Tsuruhara
>Assignee: Tomu Tsuruhara
>Priority: Trivial
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17773.master.001.patch
>
>
> Even though it's completely valid, VerifyReplication tool says "Invalid
> argument" when specifying {{\-\-recomparesleep}} or {{\-\-delimiter}} option.
> {noformat}
> $ bin/hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication
> --recomparesleep=5 1 foo
> ERROR: Invalid argument '--recomparesleep=5'
> Usage: verifyrep [--starttime=X] [--endtime=Y] [--families=A]
> [--row-prefixes=B] [--delimiter=] [--recomparesleep=] [--verbose]
>
> Options:
> starttimebeginning of the time range
> without endtime means from starttime to forever
> endtime end of the time range
> versions number of cell versions to verify
> raw includes raw scan if given in options
> families comma-separated list of families to copy
> row-prefixes comma-separated list of row key prefixes to filter on
> delimiterthe delimiter used in display around rowkey
> recomparesleep milliseconds to sleep before recompare row, default value
> is 0 which disables the recompare.
> verbose logs row keys of good rows
> Args:
> peerid Id of the peer used for verification, must match the one given
> for replication
> tablenameName of the table to verify
> Examples:
> To verify the data replicated from TestTable for a 1 hour window with peer #5
> $ hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication
> --starttime=1265875194289 --endtime=1265878794289 5 TestTable
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17778) Remove the testing code in the AsyncRequestFutureImpl
[
https://issues.apache.org/jira/browse/HBASE-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907561#comment-15907561
]
Hadoop QA commented on HBASE-17778:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 54s
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 47s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 50s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 83m 28s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 120m 9s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:e01ee2f |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12857667/HBASE-17778.branch-1.v1.patch
|
| JIRA Issue | HBASE-17778 |
[jira] [Commented] (HBASE-17765) Reviving the merge possibility in the CompactingMemStore
[
https://issues.apache.org/jira/browse/HBASE-17765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907584#comment-15907584
]
Ted Yu commented on HBASE-17765:
updateRegionSize is added to swap() as new parameter - without explanation in
@param javadoc.
{code}
if (closeSuffix && region != null) {
{code}
closeSuffix above is changed to updateRegionSize in the patch.
Can you add some comment for the change ?
> Reviving the merge possibility in the CompactingMemStore
>
>
> Key: HBASE-17765
> URL: https://issues.apache.org/jira/browse/HBASE-17765
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
> Fix For: 2.0.0
>
> Attachments: HBASE-17765-V01.patch
>
>
> According to the new performance results presented in the HBASE-16417 we see
> that the read latency of the 90th percentile of the BASIC policy is too big
> due to the need to traverse through too many segments in the pipeline. In
> this JIRA we correct the bug in the merge sizing calculations and allow
> pipeline size threshold to be a configurable parameter.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17739) BucketCache is inefficient/wasteful/dumb in its bucket allocations
[ https://issues.apache.org/jira/browse/HBASE-17739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907611#comment-15907611 ] Janos Gub commented on HBASE-17739: --- What if instead of the fix number of buckets we set up a bucket distribution between 1K and maxsize in a way to have a controlled cache loss? e.g. we would like to ensure we lose at most 10% of our cache. Then we will have an 1K bucket, then a 1.1K, 1.21K etc. In this way we will have more typesizes (for 513K we will have 66). And forget about the rule of you must have at least one bucket for each size? > BucketCache is inefficient/wasteful/dumb in its bucket allocations > -- > > Key: HBASE-17739 > URL: https://issues.apache.org/jira/browse/HBASE-17739 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack > > By default we allocate 14 buckets with sizes from 5K to 513K. If lots of heap > given over to bucketcache and say no allocattions made for a particular > bucket size, this means we have a bunch of the bucketcache that just goes > idle/unused. > For example, say heap is 100G. We'll divide it up among the sizes. If say we > only ever do 5k records, then most of the cache will go unused while the > allocation for 5k objects will see churn. > Here is an old note of [~anoop.hbase]'s' from a conversation on bucket cache > we had offlist that describes the issue: > "By default we have those 14 buckets with size range of 5K to 513K. > All sizes will have one bucket (with size 513*4) each except the > last size.. ie. 513K sized many buckets will be there. If we keep on > writing only same sized blocks, we may loose all in btw sized buckets. > Say we write only 4K sized blocks. We will 1st fill the bucket in 5K > size. There is only one such bucket. Once this is filled, we will try > to grab a complete free bucket from other sizes.. But we can not take > it from 9K... 385K sized ones as there is only ONE bucket for these > sizes. We will take only from 513 size.. There are many in that... > So we will eventually take all the buckets from 513 except the last > one.. Ya it has to keep at least one in evey size.. So we will > loose these much size.. They are of no use." > We should set the size type on the fly as the records come in. > Or better, we should choose record size on the fly. Here is another comment > from [~anoop.hbase]: > "The second is the biggest contributor. Suppose instead of 4K > sized blocks, the user has 2 K sized blocks.. When we write a block to > bucket slot, we will reserve size equal to the allocated size for that block. > So when we write 2K sized blocks (may be actual size a bit more than > 2K ) we will take 5K with each of the block. So u can see that we are > loosing ~3K with every block. Means we are loosing more than half." > He goes on: "If am 100% sure that all my table having 2K HFile block size, I > need to give this config a value 3 * 1024 (Exact 2 K if I give there may be > again problem! That is another story we need to see how we can give > more guarantee for the block size restriction HBASE-15248).. So here also > ~1K loose for every 2K.. So some thing like a 30% loose !!! :-(“" > So, we should figure the record sizes ourselves on the fly. > Anything less has us wasting loads of cache space, nvm inefficiences lost > because of how we serialize base types to cache. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17739) BucketCache is inefficient/wasteful/dumb in its bucket allocations
[ https://issues.apache.org/jira/browse/HBASE-17739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907611#comment-15907611 ] Janos Gub edited comment on HBASE-17739 at 3/13/17 3:00 PM: What if instead of the fix number of buckets we set up a bucket distribution between 1K and maxsize in a way to have a controlled cache loss? e.g. we would like to ensure we lose at most 10% of our cache. Then we will have an 1K bucket, then a 1.1K, 1.21K etc. In this way we will have more bucket sizes (for 513K we will have 66). And forget about the rule of you must have at least one bucket for each size? was (Author: gubjanos): What if instead of the fix number of buckets we set up a bucket distribution between 1K and maxsize in a way to have a controlled cache loss? e.g. we would like to ensure we lose at most 10% of our cache. Then we will have an 1K bucket, then a 1.1K, 1.21K etc. In this way we will have more typesizes (for 513K we will have 66). And forget about the rule of you must have at least one bucket for each size? > BucketCache is inefficient/wasteful/dumb in its bucket allocations > -- > > Key: HBASE-17739 > URL: https://issues.apache.org/jira/browse/HBASE-17739 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack > > By default we allocate 14 buckets with sizes from 5K to 513K. If lots of heap > given over to bucketcache and say no allocattions made for a particular > bucket size, this means we have a bunch of the bucketcache that just goes > idle/unused. > For example, say heap is 100G. We'll divide it up among the sizes. If say we > only ever do 5k records, then most of the cache will go unused while the > allocation for 5k objects will see churn. > Here is an old note of [~anoop.hbase]'s' from a conversation on bucket cache > we had offlist that describes the issue: > "By default we have those 14 buckets with size range of 5K to 513K. > All sizes will have one bucket (with size 513*4) each except the > last size.. ie. 513K sized many buckets will be there. If we keep on > writing only same sized blocks, we may loose all in btw sized buckets. > Say we write only 4K sized blocks. We will 1st fill the bucket in 5K > size. There is only one such bucket. Once this is filled, we will try > to grab a complete free bucket from other sizes.. But we can not take > it from 9K... 385K sized ones as there is only ONE bucket for these > sizes. We will take only from 513 size.. There are many in that... > So we will eventually take all the buckets from 513 except the last > one.. Ya it has to keep at least one in evey size.. So we will > loose these much size.. They are of no use." > We should set the size type on the fly as the records come in. > Or better, we should choose record size on the fly. Here is another comment > from [~anoop.hbase]: > "The second is the biggest contributor. Suppose instead of 4K > sized blocks, the user has 2 K sized blocks.. When we write a block to > bucket slot, we will reserve size equal to the allocated size for that block. > So when we write 2K sized blocks (may be actual size a bit more than > 2K ) we will take 5K with each of the block. So u can see that we are > loosing ~3K with every block. Means we are loosing more than half." > He goes on: "If am 100% sure that all my table having 2K HFile block size, I > need to give this config a value 3 * 1024 (Exact 2 K if I give there may be > again problem! That is another story we need to see how we can give > more guarantee for the block size restriction HBASE-15248).. So here also > ~1K loose for every 2K.. So some thing like a 30% loose !!! :-(“" > So, we should figure the record sizes ourselves on the fly. > Anything less has us wasting loads of cache space, nvm inefficiences lost > because of how we serialize base types to cache. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17739) BucketCache is inefficient/wasteful/dumb in its bucket allocations
[ https://issues.apache.org/jira/browse/HBASE-17739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907717#comment-15907717 ] Anoop Sam John commented on HBASE-17739: It might help with issue#2 what is mentioned in description but will badly affect the issue#1? We have more #sized buckets now and every size is supposed to keep at least one bucket, chances of many sizes going waste? Or else the sizes to be very well tuned and selected. > BucketCache is inefficient/wasteful/dumb in its bucket allocations > -- > > Key: HBASE-17739 > URL: https://issues.apache.org/jira/browse/HBASE-17739 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack > > By default we allocate 14 buckets with sizes from 5K to 513K. If lots of heap > given over to bucketcache and say no allocattions made for a particular > bucket size, this means we have a bunch of the bucketcache that just goes > idle/unused. > For example, say heap is 100G. We'll divide it up among the sizes. If say we > only ever do 5k records, then most of the cache will go unused while the > allocation for 5k objects will see churn. > Here is an old note of [~anoop.hbase]'s' from a conversation on bucket cache > we had offlist that describes the issue: > "By default we have those 14 buckets with size range of 5K to 513K. > All sizes will have one bucket (with size 513*4) each except the > last size.. ie. 513K sized many buckets will be there. If we keep on > writing only same sized blocks, we may loose all in btw sized buckets. > Say we write only 4K sized blocks. We will 1st fill the bucket in 5K > size. There is only one such bucket. Once this is filled, we will try > to grab a complete free bucket from other sizes.. But we can not take > it from 9K... 385K sized ones as there is only ONE bucket for these > sizes. We will take only from 513 size.. There are many in that... > So we will eventually take all the buckets from 513 except the last > one.. Ya it has to keep at least one in evey size.. So we will > loose these much size.. They are of no use." > We should set the size type on the fly as the records come in. > Or better, we should choose record size on the fly. Here is another comment > from [~anoop.hbase]: > "The second is the biggest contributor. Suppose instead of 4K > sized blocks, the user has 2 K sized blocks.. When we write a block to > bucket slot, we will reserve size equal to the allocated size for that block. > So when we write 2K sized blocks (may be actual size a bit more than > 2K ) we will take 5K with each of the block. So u can see that we are > loosing ~3K with every block. Means we are loosing more than half." > He goes on: "If am 100% sure that all my table having 2K HFile block size, I > need to give this config a value 3 * 1024 (Exact 2 K if I give there may be > again problem! That is another story we need to see how we can give > more guarantee for the block size restriction HBASE-15248).. So here also > ~1K loose for every 2K.. So some thing like a 30% loose !!! :-(“" > So, we should figure the record sizes ourselves on the fly. > Anything less has us wasting loads of cache space, nvm inefficiences lost > because of how we serialize base types to cache. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17773) VerifyReplication tool wrongly emits warning "ERROR: Invalid argument '--recomparesleep=xx'"
[
https://issues.apache.org/jira/browse/HBASE-17773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907733#comment-15907733
]
Hudson commented on HBASE-17773:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2665 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/2665/])
HBASE-17773 VerifyReplication tool wrongly emits Invalid arguments error
(zghao: rev fee67bcf1432cd16720fb97a0135bd67b0d2b064)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.java
> VerifyReplication tool wrongly emits warning "ERROR: Invalid argument
> '--recomparesleep=xx'"
>
>
> Key: HBASE-17773
> URL: https://issues.apache.org/jira/browse/HBASE-17773
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Tomu Tsuruhara
>Assignee: Tomu Tsuruhara
>Priority: Trivial
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17773.master.001.patch
>
>
> Even though it's completely valid, VerifyReplication tool says "Invalid
> argument" when specifying {{\-\-recomparesleep}} or {{\-\-delimiter}} option.
> {noformat}
> $ bin/hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication
> --recomparesleep=5 1 foo
> ERROR: Invalid argument '--recomparesleep=5'
> Usage: verifyrep [--starttime=X] [--endtime=Y] [--families=A]
> [--row-prefixes=B] [--delimiter=] [--recomparesleep=] [--verbose]
>
> Options:
> starttimebeginning of the time range
> without endtime means from starttime to forever
> endtime end of the time range
> versions number of cell versions to verify
> raw includes raw scan if given in options
> families comma-separated list of families to copy
> row-prefixes comma-separated list of row key prefixes to filter on
> delimiterthe delimiter used in display around rowkey
> recomparesleep milliseconds to sleep before recompare row, default value
> is 0 which disables the recompare.
> verbose logs row keys of good rows
> Args:
> peerid Id of the peer used for verification, must match the one given
> for replication
> tablenameName of the table to verify
> Examples:
> To verify the data replicated from TestTable for a 1 hour window with peer #5
> $ hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication
> --starttime=1265875194289 --endtime=1265878794289 5 TestTable
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17739) BucketCache is inefficient/wasteful/dumb in its bucket allocations
[ https://issues.apache.org/jira/browse/HBASE-17739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907736#comment-15907736 ] Janos Gub commented on HBASE-17739: --- Why is it needed to hold at least one bucket for each size? > BucketCache is inefficient/wasteful/dumb in its bucket allocations > -- > > Key: HBASE-17739 > URL: https://issues.apache.org/jira/browse/HBASE-17739 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack > > By default we allocate 14 buckets with sizes from 5K to 513K. If lots of heap > given over to bucketcache and say no allocattions made for a particular > bucket size, this means we have a bunch of the bucketcache that just goes > idle/unused. > For example, say heap is 100G. We'll divide it up among the sizes. If say we > only ever do 5k records, then most of the cache will go unused while the > allocation for 5k objects will see churn. > Here is an old note of [~anoop.hbase]'s' from a conversation on bucket cache > we had offlist that describes the issue: > "By default we have those 14 buckets with size range of 5K to 513K. > All sizes will have one bucket (with size 513*4) each except the > last size.. ie. 513K sized many buckets will be there. If we keep on > writing only same sized blocks, we may loose all in btw sized buckets. > Say we write only 4K sized blocks. We will 1st fill the bucket in 5K > size. There is only one such bucket. Once this is filled, we will try > to grab a complete free bucket from other sizes.. But we can not take > it from 9K... 385K sized ones as there is only ONE bucket for these > sizes. We will take only from 513 size.. There are many in that... > So we will eventually take all the buckets from 513 except the last > one.. Ya it has to keep at least one in evey size.. So we will > loose these much size.. They are of no use." > We should set the size type on the fly as the records come in. > Or better, we should choose record size on the fly. Here is another comment > from [~anoop.hbase]: > "The second is the biggest contributor. Suppose instead of 4K > sized blocks, the user has 2 K sized blocks.. When we write a block to > bucket slot, we will reserve size equal to the allocated size for that block. > So when we write 2K sized blocks (may be actual size a bit more than > 2K ) we will take 5K with each of the block. So u can see that we are > loosing ~3K with every block. Means we are loosing more than half." > He goes on: "If am 100% sure that all my table having 2K HFile block size, I > need to give this config a value 3 * 1024 (Exact 2 K if I give there may be > again problem! That is another story we need to see how we can give > more guarantee for the block size restriction HBASE-15248).. So here also > ~1K loose for every 2K.. So some thing like a 30% loose !!! :-(“" > So, we should figure the record sizes ourselves on the fly. > Anything less has us wasting loads of cache space, nvm inefficiences lost > because of how we serialize base types to cache. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17739) BucketCache is inefficient/wasteful/dumb in its bucket allocations
[ https://issues.apache.org/jira/browse/HBASE-17739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907743#comment-15907743 ] Anoop Sam John commented on HBASE-17739: One bucket size (Say 8 KB) is there as per the config and we dont know what time there will be a need for caching a block of this size of near this size. If there is not even one bucket of this size, we will end up in having no most apt bucket to store that block. Say that block was 7.5 KB and the next size to 8 KB bucket is 12 KB. So the selection algo has to select the 12 KB sized bucket and the wastage will be more.. Might be like, we will need more intelligence in the system so as to auto tune these sizes. That is also an aim from this jira if am not wrong.. Lets take it to next levels. > BucketCache is inefficient/wasteful/dumb in its bucket allocations > -- > > Key: HBASE-17739 > URL: https://issues.apache.org/jira/browse/HBASE-17739 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack > > By default we allocate 14 buckets with sizes from 5K to 513K. If lots of heap > given over to bucketcache and say no allocattions made for a particular > bucket size, this means we have a bunch of the bucketcache that just goes > idle/unused. > For example, say heap is 100G. We'll divide it up among the sizes. If say we > only ever do 5k records, then most of the cache will go unused while the > allocation for 5k objects will see churn. > Here is an old note of [~anoop.hbase]'s' from a conversation on bucket cache > we had offlist that describes the issue: > "By default we have those 14 buckets with size range of 5K to 513K. > All sizes will have one bucket (with size 513*4) each except the > last size.. ie. 513K sized many buckets will be there. If we keep on > writing only same sized blocks, we may loose all in btw sized buckets. > Say we write only 4K sized blocks. We will 1st fill the bucket in 5K > size. There is only one such bucket. Once this is filled, we will try > to grab a complete free bucket from other sizes.. But we can not take > it from 9K... 385K sized ones as there is only ONE bucket for these > sizes. We will take only from 513 size.. There are many in that... > So we will eventually take all the buckets from 513 except the last > one.. Ya it has to keep at least one in evey size.. So we will > loose these much size.. They are of no use." > We should set the size type on the fly as the records come in. > Or better, we should choose record size on the fly. Here is another comment > from [~anoop.hbase]: > "The second is the biggest contributor. Suppose instead of 4K > sized blocks, the user has 2 K sized blocks.. When we write a block to > bucket slot, we will reserve size equal to the allocated size for that block. > So when we write 2K sized blocks (may be actual size a bit more than > 2K ) we will take 5K with each of the block. So u can see that we are > loosing ~3K with every block. Means we are loosing more than half." > He goes on: "If am 100% sure that all my table having 2K HFile block size, I > need to give this config a value 3 * 1024 (Exact 2 K if I give there may be > again problem! That is another story we need to see how we can give > more guarantee for the block size restriction HBASE-15248).. So here also > ~1K loose for every 2K.. So some thing like a 30% loose !!! :-(“" > So, we should figure the record sizes ourselves on the fly. > Anything less has us wasting loads of cache space, nvm inefficiences lost > because of how we serialize base types to cache. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17747) Support both weak and soft object pool
[
https://issues.apache.org/jira/browse/HBASE-17747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907751#comment-15907751
]
Anoop Sam John commented on HBASE-17747:
That is good Yu Li.
bq.Support both weak and soft object pool
So here we change it from Weak to Soft ref pool. Can u change the jira Title
to be clear?
> Support both weak and soft object pool
> --
>
> Key: HBASE-17747
> URL: https://issues.apache.org/jira/browse/HBASE-17747
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17747.patch, HBASE-17747.v2.patch,
> HBASE-17747.v3.patch
>
>
> During YCSB testing on embedded mode after HBASE-17744, we found that under
> high read load GC is quite severe even with offheap L2 cache. After some
> investigation, we found it's caused by using weak reference in
> {{IdReadWriteLock}}. In embedded mode the read is so quick that the lock
> might already get promoted to the old generation when the weak reference is
> cleared, which causes dirty card table (old reference get removed and new
> lock object set into {{referenceCache}}, see {{WeakObjectPool#get}}) thus
> slowing YGC. In distributed mode there'll also be more lock object created
> with weak reference than soft reference that slowing down the processing.
> So we proposed to use soft reference for this {{IdReadWriteLock}} used in
> cache, which won't get cleared until JVM memory is not enough, and could
> resolve the issue mentioned above. What's more, we propose to extend the
> {{WeakObjectPool}} to be more generate to support both weak and soft
> reference.
> Note that the GC issue only emerges under embedded mode with DirectOperator,
> in which case all costs on the wire is removed thus produces extremely high
> concurrency.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17777) TestMemstoreLAB#testLABThreading runs too long for a small test
[ https://issues.apache.org/jira/browse/HBASE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907758#comment-15907758 ] Anoop Sam John commented on HBASE-1: With out reducing the #threads and chunk, if we can reduce the test timing just by reducing the cell's size, then it should be done. Hope there is no need to reduce #cells written too. Good > TestMemstoreLAB#testLABThreading runs too long for a small test > --- > > Key: HBASE-1 > URL: https://issues.apache.org/jira/browse/HBASE-1 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > > While working on ChunkCreator/ChunkMap found that the test in > TestMSLAB#testLABThreading() runs for almost 5 mins and the whole test is > under smallTest category. > The reason is that we are creating 35*2MB chunks from MSLAB. We try writing > data to these chunks until they are 50MB in size. > And while verifying in order to check if the chunks are not > overwritten/overlapped we verify the content of the buffers. > So we actually keep comparing 50MB buffer n number of times. I suggest we > change this in a way that at max we create chunks whose size is totally at > 1MB or may be even lesser and write cells which are smaller in size. By doing > this we can drastically reduce the run time of this test. May be something > less than 1 min. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907770#comment-15907770 ] ramkrishna.s.vasudevan commented on HBASE-15314: [~zjushch] There was one more comment in RB. Are you going to add a new patch here fixing it? If there is a consensus we can commit this and close this JIRA. > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, > HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, > HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch, > HBASE-15314-v6.patch, HBASE-15314-v7.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15143) Procedure v2 - Web UI displaying queues
[
https://issues.apache.org/jira/browse/HBASE-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907768#comment-15907768
]
Hadoop QA commented on HBASE-15143:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 1s
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 1s
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 30s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 57s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
3s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 55s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 21m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
1s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 36 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s
{color} | {color:red} The patch 13 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 34s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 26s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 53s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 42s
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 12s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 94m 30s
{color} | {color:green} hbase-server in the patch passed. {colo
[jira] [Commented] (HBASE-17766) Generate Javadoc for hbase-spark module
[ https://issues.apache.org/jira/browse/HBASE-17766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907818#comment-15907818 ] Yi Liang commented on HBASE-17766: -- @stack Thanks for the suggestion, will submit the patch soon. > Generate Javadoc for hbase-spark module > > > Key: HBASE-17766 > URL: https://issues.apache.org/jira/browse/HBASE-17766 > Project: HBase > Issue Type: Bug >Reporter: Yi Liang >Assignee: Yi Liang > Fix For: 2.0.0 > > > Scala classes in hbase-spark module aren't showing up in our API docs nor > our internal API docs. see https://hbase.apache.org/apidocs/ -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17501) NullPointerException after Datanodes Decommissioned and Terminated
[ https://issues.apache.org/jira/browse/HBASE-17501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907912#comment-15907912 ] James Moore commented on HBASE-17501: - Looks like QA tests are passing - are we good to merge? cheers, --James > NullPointerException after Datanodes Decommissioned and Terminated > -- > > Key: HBASE-17501 > URL: https://issues.apache.org/jira/browse/HBASE-17501 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 > Environment: CentOS Derivative with a derivative of the 3.18.43 > kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches. >Reporter: Patrick Dignan >Assignee: James Moore >Priority: Minor > Attachments: HBASE_17501.patch, HBASE_17501.patch, > HBASE_17501.patch.v2, HBASE_17501.patch.v3, HBASE_17501.patch.v4, > HBASE_17501.v3 > > > We recently encountered an interesting NullPointerException in HDFS that > bubbles up to HBase, and is resolved be restarting the regionserver. The > issue was exhibited while we were replacing a set of nodes in one of our > clusters with a new set. We did the following: > 1. Turn off the HBase balancer > 2. Gracefully move the regions off the nodes we’re shutting off using a tool > we wrote to do so > 3. Decommission the datanodes using the HDFS exclude hosts file and hdfs > dfsadmin -refreshNodes > 4. Wait for the datanodes to decommission fully > 5. Terminate the VMs the instances are running inside. > A few notes. We did not shutdown the datanode processes, and the nodes were > therefore not marked as dead by the namenode. We simply terminated the > datanode VM (in this case an AWS instance). The nodes were marked as > decommissioned. We are running our clusters with DNS, and when we terminate > VMs, the associated CName is removed and no longer resolves. The errors do > not seem to resolve without a restart. > After we did this, the remaining regionservers started throwing > NullPointerExceptions with the following stack trace: > 2017-01-19 23:09:05,638 DEBUG org.apache.hadoop.hbase.ipc.RpcServer: > RpcServer.RW.fifo.Q.read.handler=80,queue=14,port=60020: callId: 1727723891 > service: ClientService methodName: Scan size: 216 connection: > 172.16.36.128:31538 > java.io.IOException > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2214) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:204) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183) > Caused by: java.lang.NullPointerException > at org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1564) > at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:62) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1434) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1682) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1542) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:445) > at > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:266) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:642) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:592) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:294) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:199) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:343) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:198) > at > org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2106) > at > org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2096) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5544) > at > org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2569) > at > org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2555) > at > org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2536) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2405) > at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33738) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServ
[jira] [Updated] (HBASE-17723) ClientAsyncPrefetchScanner may end prematurely when the size of the cache is one
[ https://issues.apache.org/jira/browse/HBASE-17723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17723: --- Status: Open (was: Patch Available) QA disappeared... retry > ClientAsyncPrefetchScanner may end prematurely when the size of the cache is > one > > > Key: HBASE-17723 > URL: https://issues.apache.org/jira/browse/HBASE-17723 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 > Environment: Ubuntu 14.04 LTS / x86_64 / 16GB RAM >Reporter: Anup Halarnkar >Assignee: Chia-Ping Tsai > Fix For: 2.0.0 > > Attachments: HBASE-17723.v0.patch, HBASE-17723.v1.patch, > HBASE-17723.v1.patch, HBASE-17723.v1.patch, HBASE-17723.v1.patch > > > Command to build and test: > mvn clean install -X > JAVA_TOOL_OPTIONS: -Xmx4G -XX:MaxPermSize=4G -Xms1G > Failure in tests: > Results : > Failed tests: > > TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync:161->testRowsSeenMetric:174->testRowsSeenMetric:204->testMetric:330 > Metric: ROWS_SCANNED Expected: 9 Actual: 8 expected:<9> but was:<8> > > TestRecoveredEdits.testReplayWorksThoughLotsOfFlushing:78->testReplayWorksWithMemoryCompactionPolicy:149->verifyAllEditsMadeItIn:195 > Failed to find > W\xE1X\xDBv\xD8}\x07\xBF\xC1I\xF1\xF5\xB9\x84\xD8/meta:prev/1422491946444/Put/vlen=16/seqid=0 > Tests run: 1761, Failures: 2, Errors: 0, Skipped: 6 > > This is the output for 1st test failure: > --- > Test set: org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > --- > Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.971 sec <<< > FAILURE! - in org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > testRowsSeenMetricWithAsync(org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide) > Time elapsed: 1.134 sec <<< FAILURE! > java.lang.AssertionError: Metric: ROWS_SCANNED Expected: 9 Actual: 8 > expected:<9> but was:<8> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testMetric(TestServerSideScanMetricsFromClientSide.java:330) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetric(TestServerSideScanMetricsFromClientSide.java:204) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetric(TestServerSideScanMetricsFromClientSide.java:174) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync(TestServerSideScanMetricsFromClientSide.java:161) > > This is output of 2nd test failure: > --- > Test set: org.apache.hadoop.hbase.regionserver.TestRecoveredEdits > --- > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 11.911 sec > <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestRecoveredEdits > testReplayWorksThoughLotsOfFlushing(org.apache.hadoop.hbase.regionserver.TestRecoveredEdits) > Time elapsed: 11.871 sec <<< FAILURE! > java.lang.AssertionError: Failed to find > W\xE1X\xDBv\xD8}\x07\xBF\xC1I\xF1\xF5\xB9\x84\xD8/meta:prev/1422491946444/Put/vlen=16/seqid=0 > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.verifyAllEditsMadeItIn(TestRecoveredEdits.java:195) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.testReplayWorksWithMemoryCompactionPolicy(TestRecoveredEdits.java:149) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.testReplayWorksThoughLotsOfFlushing(TestRecoveredEdits.java:78) > Please let me know if you need more information > Thanks, > Anup -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17723) ClientAsyncPrefetchScanner may end prematurely when the size of the cache is one
[ https://issues.apache.org/jira/browse/HBASE-17723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17723: --- Attachment: HBASE-17723.v1.patch > ClientAsyncPrefetchScanner may end prematurely when the size of the cache is > one > > > Key: HBASE-17723 > URL: https://issues.apache.org/jira/browse/HBASE-17723 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 > Environment: Ubuntu 14.04 LTS / x86_64 / 16GB RAM >Reporter: Anup Halarnkar >Assignee: Chia-Ping Tsai > Fix For: 2.0.0 > > Attachments: HBASE-17723.v0.patch, HBASE-17723.v1.patch, > HBASE-17723.v1.patch, HBASE-17723.v1.patch, HBASE-17723.v1.patch, > HBASE-17723.v1.patch > > > Command to build and test: > mvn clean install -X > JAVA_TOOL_OPTIONS: -Xmx4G -XX:MaxPermSize=4G -Xms1G > Failure in tests: > Results : > Failed tests: > > TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync:161->testRowsSeenMetric:174->testRowsSeenMetric:204->testMetric:330 > Metric: ROWS_SCANNED Expected: 9 Actual: 8 expected:<9> but was:<8> > > TestRecoveredEdits.testReplayWorksThoughLotsOfFlushing:78->testReplayWorksWithMemoryCompactionPolicy:149->verifyAllEditsMadeItIn:195 > Failed to find > W\xE1X\xDBv\xD8}\x07\xBF\xC1I\xF1\xF5\xB9\x84\xD8/meta:prev/1422491946444/Put/vlen=16/seqid=0 > Tests run: 1761, Failures: 2, Errors: 0, Skipped: 6 > > This is the output for 1st test failure: > --- > Test set: org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > --- > Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.971 sec <<< > FAILURE! - in org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > testRowsSeenMetricWithAsync(org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide) > Time elapsed: 1.134 sec <<< FAILURE! > java.lang.AssertionError: Metric: ROWS_SCANNED Expected: 9 Actual: 8 > expected:<9> but was:<8> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testMetric(TestServerSideScanMetricsFromClientSide.java:330) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetric(TestServerSideScanMetricsFromClientSide.java:204) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetric(TestServerSideScanMetricsFromClientSide.java:174) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync(TestServerSideScanMetricsFromClientSide.java:161) > > This is output of 2nd test failure: > --- > Test set: org.apache.hadoop.hbase.regionserver.TestRecoveredEdits > --- > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 11.911 sec > <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestRecoveredEdits > testReplayWorksThoughLotsOfFlushing(org.apache.hadoop.hbase.regionserver.TestRecoveredEdits) > Time elapsed: 11.871 sec <<< FAILURE! > java.lang.AssertionError: Failed to find > W\xE1X\xDBv\xD8}\x07\xBF\xC1I\xF1\xF5\xB9\x84\xD8/meta:prev/1422491946444/Put/vlen=16/seqid=0 > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.verifyAllEditsMadeItIn(TestRecoveredEdits.java:195) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.testReplayWorksWithMemoryCompactionPolicy(TestRecoveredEdits.java:149) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.testReplayWorksThoughLotsOfFlushing(TestRecoveredEdits.java:78) > Please let me know if you need more information > Thanks, > Anup -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17723) ClientAsyncPrefetchScanner may end prematurely when the size of the cache is one
[ https://issues.apache.org/jira/browse/HBASE-17723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-17723: --- Status: Patch Available (was: Open) > ClientAsyncPrefetchScanner may end prematurely when the size of the cache is > one > > > Key: HBASE-17723 > URL: https://issues.apache.org/jira/browse/HBASE-17723 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 > Environment: Ubuntu 14.04 LTS / x86_64 / 16GB RAM >Reporter: Anup Halarnkar >Assignee: Chia-Ping Tsai > Fix For: 2.0.0 > > Attachments: HBASE-17723.v0.patch, HBASE-17723.v1.patch, > HBASE-17723.v1.patch, HBASE-17723.v1.patch, HBASE-17723.v1.patch, > HBASE-17723.v1.patch > > > Command to build and test: > mvn clean install -X > JAVA_TOOL_OPTIONS: -Xmx4G -XX:MaxPermSize=4G -Xms1G > Failure in tests: > Results : > Failed tests: > > TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync:161->testRowsSeenMetric:174->testRowsSeenMetric:204->testMetric:330 > Metric: ROWS_SCANNED Expected: 9 Actual: 8 expected:<9> but was:<8> > > TestRecoveredEdits.testReplayWorksThoughLotsOfFlushing:78->testReplayWorksWithMemoryCompactionPolicy:149->verifyAllEditsMadeItIn:195 > Failed to find > W\xE1X\xDBv\xD8}\x07\xBF\xC1I\xF1\xF5\xB9\x84\xD8/meta:prev/1422491946444/Put/vlen=16/seqid=0 > Tests run: 1761, Failures: 2, Errors: 0, Skipped: 6 > > This is the output for 1st test failure: > --- > Test set: org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > --- > Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.971 sec <<< > FAILURE! - in org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > testRowsSeenMetricWithAsync(org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide) > Time elapsed: 1.134 sec <<< FAILURE! > java.lang.AssertionError: Metric: ROWS_SCANNED Expected: 9 Actual: 8 > expected:<9> but was:<8> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testMetric(TestServerSideScanMetricsFromClientSide.java:330) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetric(TestServerSideScanMetricsFromClientSide.java:204) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetric(TestServerSideScanMetricsFromClientSide.java:174) > at > org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide.testRowsSeenMetricWithAsync(TestServerSideScanMetricsFromClientSide.java:161) > > This is output of 2nd test failure: > --- > Test set: org.apache.hadoop.hbase.regionserver.TestRecoveredEdits > --- > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 11.911 sec > <<< FAILURE! - in org.apache.hadoop.hbase.regionserver.TestRecoveredEdits > testReplayWorksThoughLotsOfFlushing(org.apache.hadoop.hbase.regionserver.TestRecoveredEdits) > Time elapsed: 11.871 sec <<< FAILURE! > java.lang.AssertionError: Failed to find > W\xE1X\xDBv\xD8}\x07\xBF\xC1I\xF1\xF5\xB9\x84\xD8/meta:prev/1422491946444/Put/vlen=16/seqid=0 > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.verifyAllEditsMadeItIn(TestRecoveredEdits.java:195) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.testReplayWorksWithMemoryCompactionPolicy(TestRecoveredEdits.java:149) > at > org.apache.hadoop.hbase.regionserver.TestRecoveredEdits.testReplayWorksThoughLotsOfFlushing(TestRecoveredEdits.java:78) > Please let me know if you need more information > Thanks, > Anup -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17739) BucketCache is inefficient/wasteful/dumb in its bucket allocations
[ https://issues.apache.org/jira/browse/HBASE-17739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15922507#comment-15922507 ] Janos Gub commented on HBASE-17739: --- If there is no bucket of that size we will end up evicting a bucket and initializing a correct sized one right? (That is how we avoid having almost only 513K sized buckets with current implementation) > BucketCache is inefficient/wasteful/dumb in its bucket allocations > -- > > Key: HBASE-17739 > URL: https://issues.apache.org/jira/browse/HBASE-17739 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack > > By default we allocate 14 buckets with sizes from 5K to 513K. If lots of heap > given over to bucketcache and say no allocattions made for a particular > bucket size, this means we have a bunch of the bucketcache that just goes > idle/unused. > For example, say heap is 100G. We'll divide it up among the sizes. If say we > only ever do 5k records, then most of the cache will go unused while the > allocation for 5k objects will see churn. > Here is an old note of [~anoop.hbase]'s' from a conversation on bucket cache > we had offlist that describes the issue: > "By default we have those 14 buckets with size range of 5K to 513K. > All sizes will have one bucket (with size 513*4) each except the > last size.. ie. 513K sized many buckets will be there. If we keep on > writing only same sized blocks, we may loose all in btw sized buckets. > Say we write only 4K sized blocks. We will 1st fill the bucket in 5K > size. There is only one such bucket. Once this is filled, we will try > to grab a complete free bucket from other sizes.. But we can not take > it from 9K... 385K sized ones as there is only ONE bucket for these > sizes. We will take only from 513 size.. There are many in that... > So we will eventually take all the buckets from 513 except the last > one.. Ya it has to keep at least one in evey size.. So we will > loose these much size.. They are of no use." > We should set the size type on the fly as the records come in. > Or better, we should choose record size on the fly. Here is another comment > from [~anoop.hbase]: > "The second is the biggest contributor. Suppose instead of 4K > sized blocks, the user has 2 K sized blocks.. When we write a block to > bucket slot, we will reserve size equal to the allocated size for that block. > So when we write 2K sized blocks (may be actual size a bit more than > 2K ) we will take 5K with each of the block. So u can see that we are > loosing ~3K with every block. Means we are loosing more than half." > He goes on: "If am 100% sure that all my table having 2K HFile block size, I > need to give this config a value 3 * 1024 (Exact 2 K if I give there may be > again problem! That is another story we need to see how we can give > more guarantee for the block size restriction HBASE-15248).. So here also > ~1K loose for every 2K.. So some thing like a 30% loose !!! :-(“" > So, we should figure the record sizes ourselves on the fly. > Anything less has us wasting loads of cache space, nvm inefficiences lost > because of how we serialize base types to cache. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15597) Clean up configuration keys used in hbase-spark module
[ https://issues.apache.org/jira/browse/HBASE-15597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He updated HBASE-15597: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the contribution and review. Pushed to master. > Clean up configuration keys used in hbase-spark module > -- > > Key: HBASE-15597 > URL: https://issues.apache.org/jira/browse/HBASE-15597 > Project: HBase > Issue Type: Bug > Components: spark >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Yi Liang >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-15597-v1.patch, HBASE-15597-V2.patch, > HBASE-15597-V3.patch, HBase-15597-V4.patch > > > This should be considered a blocker for backport to branch-1 since it will > impact our compatibility. > The constants we expose in configuration should all start with "hbase". Since > our configurations keys for the spark integration all relate to that system, > the prefix for all configuration keys (excluding those cases where we need to > do something special due to restrictions in how properties are handled by > e.g. spark) should be "hbase.spark". > Before publishing a public api labeled version of our spark integration we > should review all of our configuration keys to make sure they either conform > to the "hbase.spark" prefix or they have a comment documenting why they need > to be otherwise. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14161) Add hbase-spark integration tests to IT jenkins job
[ https://issues.apache.org/jira/browse/HBASE-14161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15922746#comment-15922746 ] Jerry He commented on HBASE-14161: -- Let me take a look. Maybe I misunderstood the JIRA. > Add hbase-spark integration tests to IT jenkins job > --- > > Key: HBASE-14161 > URL: https://issues.apache.org/jira/browse/HBASE-14161 > Project: HBase > Issue Type: Task > Components: build >Reporter: Sean Busbey > Fix For: 2.0.0 > > > expand the set of ITs we run to include the new hbase-spark tests. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15922780#comment-15922780 ] Jerry He commented on HBASE-16179: -- Hi, Ted I also meant the binary tarball assembly. I did with the v23 patch: mvn package install assembly:single -DskipTests I could not see any hbase-spark jars or the compat jars in the lib dir. We need one and only version of the jars in the binary tarball. > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v10.txt, 16179.v11.txt, > 16179.v12.txt, 16179.v12.txt, 16179.v12.txt, 16179.v13.txt, 16179.v15.txt, > 16179.v16.txt, 16179.v18.txt, 16179.v19.txt, 16179.v19.txt, 16179.v1.txt, > 16179.v1.txt, 16179.v20.txt, 16179.v22.txt, 16179.v23.txt, 16179.v4.txt, > 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16179: --- Attachment: 16179.v24.txt Patch v24 is rebased on master branch. Will look into what Jerry found. > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v10.txt, 16179.v11.txt, > 16179.v12.txt, 16179.v12.txt, 16179.v12.txt, 16179.v13.txt, 16179.v15.txt, > 16179.v16.txt, 16179.v18.txt, 16179.v19.txt, 16179.v19.txt, 16179.v1.txt, > 16179.v1.txt, 16179.v20.txt, 16179.v22.txt, 16179.v23.txt, 16179.v24.txt, > 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17779) disable_table_replication returns misleading message and does not turn off repliaton
[ https://issues.apache.org/jira/browse/HBASE-17779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15922886#comment-15922886 ] Ted Yu commented on HBASE-17779: [~Bhupendra]: What do you think of the patch ? > disable_table_replication returns misleading message and does not turn off > repliaton > > > Key: HBASE-17779 > URL: https://issues.apache.org/jira/browse/HBASE-17779 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Janos Gub > Attachments: HBASE-17779.patch > > > Currently HBaseAdmin#isTableRepEnabled() returns false if ANY of the columns > families is not replicated. > Because of this if you have a table where replication is partially enabled, > calling disable_table_replication will not have any effect, but will report > that replication for a given table is disabled. > Workaround is enabling table replication before disabling. > As a solution quoted from HBASE-17460: > Admin#disableTableReplication() returns nothing and isTableRepEnabled() > returns boolean. For master branch, we can let isTableRepEnabled() return > enum (partially disabled, disabled, etc) This way, > Admin#disableTableReplication() can return meaningful value to the user. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15922885#comment-15922885 ] Jerry He commented on HBASE-16179: -- Adding to the argument what spark/scala version combinations we want to support out of the box, it does not seem practical that we have four binary distribution tarballs. > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v10.txt, 16179.v11.txt, > 16179.v12.txt, 16179.v12.txt, 16179.v12.txt, 16179.v13.txt, 16179.v15.txt, > 16179.v16.txt, 16179.v18.txt, 16179.v19.txt, 16179.v19.txt, 16179.v1.txt, > 16179.v1.txt, 16179.v20.txt, 16179.v22.txt, 16179.v23.txt, 16179.v24.txt, > 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15922888#comment-15922888 ] Jerry He commented on HBASE-16179: -- The SparkSQLPushDownFilter makes the hbase-spark jar necessary in hbase lib. > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v10.txt, 16179.v11.txt, > 16179.v12.txt, 16179.v12.txt, 16179.v12.txt, 16179.v13.txt, 16179.v15.txt, > 16179.v16.txt, 16179.v18.txt, 16179.v19.txt, 16179.v19.txt, 16179.v1.txt, > 16179.v1.txt, 16179.v20.txt, 16179.v22.txt, 16179.v23.txt, 16179.v24.txt, > 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17723) ClientAsyncPrefetchScanner may end prematurely when the size of the cache is one
[
https://issues.apache.org/jira/browse/HBASE-17723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15922908#comment-15922908
]
Hadoop QA commented on HBASE-17723:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 55s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 21s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 102m 37s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 152m 31s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12858204/HBASE-17723.v1.patch |
| JIRA Issue | HBASE-17723 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux acdbaa2807eb 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / fee67bc |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6081/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6081/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> ClientAsyncPrefetchScanner may en
[jira] [Reopened] (HBASE-17771) [C++] Classes required for implementation of BatchCallerBuilder
[ https://issues.apache.org/jira/browse/HBASE-17771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar reopened HBASE-17771: --- Accidentally resolved this. Reopening. > [C++] Classes required for implementation of BatchCallerBuilder > --- > > Key: HBASE-17771 > URL: https://issues.apache.org/jira/browse/HBASE-17771 > Project: HBase > Issue Type: Sub-task >Reporter: Sudeep Sunthankar >Assignee: Sudeep Sunthankar > Fix For: HBASE-14850 > > Attachments: HBASE-17771.HBASE-14850.v1.patch > > > Separating depedencies of BatchCallerBuilder. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HBASE-17771) [C++] Classes required for implementation of BatchCallerBuilder
[ https://issues.apache.org/jira/browse/HBASE-17771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar resolved HBASE-17771. --- Resolution: Fixed Fix Version/s: HBASE-14850 > [C++] Classes required for implementation of BatchCallerBuilder > --- > > Key: HBASE-17771 > URL: https://issues.apache.org/jira/browse/HBASE-17771 > Project: HBase > Issue Type: Sub-task >Reporter: Sudeep Sunthankar >Assignee: Sudeep Sunthankar > Fix For: HBASE-14850 > > Attachments: HBASE-17771.HBASE-14850.v1.patch > > > Separating depedencies of BatchCallerBuilder. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HBASE-17466) [C++] Test cleanup
[ https://issues.apache.org/jira/browse/HBASE-17466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar resolved HBASE-17466. --- Resolution: Fixed Fix Version/s: HBASE-14850 Pushed this to the branch. > [C++] Test cleanup > -- > > Key: HBASE-17466 > URL: https://issues.apache.org/jira/browse/HBASE-17466 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: HBASE-14850 > > Attachments: hbase-17466_v1.patch > > > The tests takes too long due to sleep and starting / stopping the cluster. We > can do some speed up. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15922999#comment-15922999 ] Ted Yu commented on HBASE-16179: hbase-assembly/src/main/assembly/hadoop-two-compat.xml needs to be modified for including the default combination for spark module. Waiting for Sean's input before making the next patch. > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v10.txt, 16179.v11.txt, > 16179.v12.txt, 16179.v12.txt, 16179.v12.txt, 16179.v13.txt, 16179.v15.txt, > 16179.v16.txt, 16179.v18.txt, 16179.v19.txt, 16179.v19.txt, 16179.v1.txt, > 16179.v1.txt, 16179.v20.txt, 16179.v22.txt, 16179.v23.txt, 16179.v24.txt, > 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17771) [C++] Classes required for implementation of BatchCallerBuilder
[
https://issues.apache.org/jira/browse/HBASE-17771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15923033#comment-15923033
]
Enis Soztutar commented on HBASE-17771:
---
- Changes in {{core/BUCK}} undos the previous changes from HBASE-17728. This is
probably due to rebasing, but we need HBASE-17728.
- {{Action::OriginalIndex()}} should be named {{Action::original_index()}} and
{{Action::GetAction()}} should be {{Action::action()}}. Maybe in the other
classes as well (my reading of the conventions is that if the class method is
an accessor to a class field, you can name it like foo_bar(), instead of
FooBar().
- Class {{MultiAction}} is not needed?. Seems to be coming from the
AsyncProcess based impl. I've checked the patch at HBASE-17576, not used there
as well.
- Remove commented out code:
{code}
+ /*static std::shared_ptr GetResults(const
std::shared_ptr& req,
+ std::unique_ptr resp);*/
The non-commented out version takes Request by raw-pointer which is probably
not what we want.
{code}
- The CellScanner::Advance() might return false as well for the below code:
{code}
-while (cell_scanner->Advance()) {
+int num_cells = 0;
+while (num_cells != result.associated_cell_count()) {
+ cell_scanner->Advance();
{code}
So, I think you should check for that, and throw an exception if we wanted to
read that many cells, but the CellScanner::Advance() is returning false.
- Are you gonna address this? {{+// TODO Revisit this class once}}
- RegionRequest::ActionList is a shared_ptr, while
ServerRequest::ActionsByRegion is not. I think you don't need the shared_ptr in
RegionRequest::ActionList, especially because you are returning via {{const
&}}.
- RequestConverter::ToGet() is good, but it is duplicating the code. Maybe make
it so that ToGetRequest() uses that.
> [C++] Classes required for implementation of BatchCallerBuilder
> ---
>
> Key: HBASE-17771
> URL: https://issues.apache.org/jira/browse/HBASE-17771
> Project: HBase
> Issue Type: Sub-task
>Reporter: Sudeep Sunthankar
>Assignee: Sudeep Sunthankar
> Fix For: HBASE-14850
>
> Attachments: HBASE-17771.HBASE-14850.v1.patch
>
>
> Separating depedencies of BatchCallerBuilder.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17501) NullPointerException after Datanodes Decommissioned and Terminated
[ https://issues.apache.org/jira/browse/HBASE-17501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17501: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.2.6 1.3.1 1.4.0 2.0.0 Status: Resolved (was: Patch Available) Pushed to branch-1.2+. Thanks for the ping [~lumost] and thanks for the patch > NullPointerException after Datanodes Decommissioned and Terminated > -- > > Key: HBASE-17501 > URL: https://issues.apache.org/jira/browse/HBASE-17501 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 > Environment: CentOS Derivative with a derivative of the 3.18.43 > kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches. >Reporter: Patrick Dignan >Assignee: James Moore >Priority: Minor > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.6 > > Attachments: HBASE_17501.patch, HBASE_17501.patch, > HBASE_17501.patch.v2, HBASE_17501.patch.v3, HBASE_17501.patch.v4, > HBASE_17501.v3 > > > We recently encountered an interesting NullPointerException in HDFS that > bubbles up to HBase, and is resolved be restarting the regionserver. The > issue was exhibited while we were replacing a set of nodes in one of our > clusters with a new set. We did the following: > 1. Turn off the HBase balancer > 2. Gracefully move the regions off the nodes we’re shutting off using a tool > we wrote to do so > 3. Decommission the datanodes using the HDFS exclude hosts file and hdfs > dfsadmin -refreshNodes > 4. Wait for the datanodes to decommission fully > 5. Terminate the VMs the instances are running inside. > A few notes. We did not shutdown the datanode processes, and the nodes were > therefore not marked as dead by the namenode. We simply terminated the > datanode VM (in this case an AWS instance). The nodes were marked as > decommissioned. We are running our clusters with DNS, and when we terminate > VMs, the associated CName is removed and no longer resolves. The errors do > not seem to resolve without a restart. > After we did this, the remaining regionservers started throwing > NullPointerExceptions with the following stack trace: > 2017-01-19 23:09:05,638 DEBUG org.apache.hadoop.hbase.ipc.RpcServer: > RpcServer.RW.fifo.Q.read.handler=80,queue=14,port=60020: callId: 1727723891 > service: ClientService methodName: Scan size: 216 connection: > 172.16.36.128:31538 > java.io.IOException > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2214) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:204) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183) > Caused by: java.lang.NullPointerException > at org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1564) > at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:62) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1434) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1682) > at > org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1542) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:445) > at > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:266) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:642) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:592) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:294) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:199) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:343) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:198) > at > org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2106) > at > org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2096) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5544) > at > org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2569) > at > org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2555) > at > org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2536) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2
[jira] [Commented] (HBASE-15143) Procedure v2 - Web UI displaying queues
[ https://issues.apache.org/jira/browse/HBASE-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15923066#comment-15923066 ] stack commented on HBASE-15143: --- It looks good [~balazs.meszaros] Is the test failure yours? Thanks for the screen shots. LockInfo[] listLocks() matches listProcedures. Good. We have to do this: 223 224 org.apache.hbase 225 hbase-protocol-shaded 226 Hmm Was trying to have it so hbase-common did NOT depend on hbase-protocol* but I see the procedure stuff already does this. We can file a follow-on JIRA to purge protos from hbase-common. Not your fault. But let me ask, you are modeling on the ProcedureInfo? if so, that is a bit of a hack. Dang. Would be good to do a right-soln for ProcedureInfo and therefore for LockInfo. But this is not your issue. Should WaitingProcedure be a ProcedureInfo altogether? OR a subclass? Or have-a PI? Could you have LockInfo protobuf use PI protobuf? LockInfo needs a license. See other source files. And TestLockUtil needs license. Maybe should be in hbase.procedure2 package? Ditto for LockUtil LockUtil and LockInfo are in the top-level of the hbase package. Is that right? I suppose PI is and you are following that precedent. Otherwise, patch looks great. > Procedure v2 - Web UI displaying queues > --- > > Key: HBASE-15143 > URL: https://issues.apache.org/jira/browse/HBASE-15143 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, UI >Reporter: Matteo Bertozzi >Assignee: Balazs Meszaros >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15143-BM-0001.patch, HBASE-15143-BM-0002.patch, > HBASE-15143-BM-0003.patch, HBASE-15143-BM-0004.patch, screenshot.png > > > We can query MasterProcedureScheduler to display the various procedures and > who is holding table/region locks. > Each procedure is in a TableQueue or ServerQueue, so it is easy to display > the procedures in its own group. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17768) [C++] Makefile should recompile only the changed sources
[ https://issues.apache.org/jira/browse/HBASE-17768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudeep Sunthankar updated HBASE-17768: -- Attachment: HBASE-17768.HBASE-14850.v2.patch # Removing the script dependency to create PB srcs # Updated help in Makefile and Makefile.protos Thanks > [C++] Makefile should recompile only the changed sources > > > Key: HBASE-17768 > URL: https://issues.apache.org/jira/browse/HBASE-17768 > Project: HBase > Issue Type: Sub-task >Reporter: Sudeep Sunthankar >Assignee: Sudeep Sunthankar > Attachments: HBASE-17768.HBASE-14850.v1.patch, > HBASE-17768.HBASE-14850.v2.patch > > > Current Makefile builds everything including PB sources everytime. This > patch addresses the issue -- This message was sent by Atlassian JIRA (v6.3.15#6346)
