[jira] [Commented] (HBASE-18055) HBASE-17917 closes the scanners while a scan is in progess for switching over to stream reads
[ https://issues.apache.org/jira/browse/HBASE-18055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011849#comment-16011849 ] Anoop Sam John commented on HBASE-18055: Here the issue is the results is not yet serialized back to client. They refer to the blocks which where there in L2 cache. Before sending reply back, the scanner seems closed and that was reducing the ref count and possibly making the ref count to 0 and evictions. > HBASE-17917 closes the scanners while a scan is in progess for switching over > to stream reads > - > > Key: HBASE-18055 > URL: https://issues.apache.org/jira/browse/HBASE-18055 > Project: HBase > Issue Type: Bug > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > > In HBASE-17917 tries to switch from pread to stream read when a specific size > of bytes are read. So in order to switch over, it closes the existing > scanners and creates a new scanners with pread=false. > When we close the exisitng scanners - if the blocks are served from offheap > cache we will decrement the ref count on those blocks and if it becomes zero > we make the block ready for eviction. Then there is a chance that the result > could be corrupted if new blocks occupy the cache. So the expectation was > that till the RPC call completes the response we will hold on to the blocks > that are referred by the scan. (except the last one). So trying to switch > over to stream read will break this expectation and hence > TestBlockEvictionfromclient fails. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-18055) HBASE-17917 closes the scanners while a scan is in progess for switching over to stream reads
[ https://issues.apache.org/jira/browse/HBASE-18055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011847#comment-16011847 ] ramkrishna.s.vasudevan edited comment on HBASE-18055 at 5/16/17 6:57 AM: - Let me try to explain. For simplicity sake say we have one row per block. We have totally 4 rows (4 blocks). All are cached in L2. The scan is reading 3 blocks (3 rows). Tries with the next next() call. Sees that the preadMaxBytes is reached and creates new scanners and seeks to the 3rd row (since row 3 was latest). And then closes all the older scanners. So the block 1 and block 2 (of row 1 and row2) are now decremented and ref count is 0 since older scanners refers to these blocks. block 3 would have been incremented to 2 (while seeking) and then while closing it would have become 1. But since block 1 or block 2 are already decremented to 0, it means any other new blocks that gets cached during this time (by other scan) can occupy the L2 cache and so the cells that are added in the Result are now corrupted because the actual blocks in the L2 cache are overwritten. was (Author: ram_krish): Let me try to explain. For simplicity sake say we have one row per block. We have totally 4 rows (4 blocks). All are cached in L2. The scan is reading 3 blocks (3 rows). Tries with the next next() call. Sees that the preadMaxBytes is reached and creates new scanners and seeks to the 3rd row (since row 3 was latest). And then closes all the older scanners. So the block 1 and block 2 (of row 1 and row2) are now decremented and ref count is 0 since older scanners refers to these blocks. block 3 would have been incremented to 2 and then while closing it would have become 1. But since block 1 or block 2 are already decremented to 0, it means any other new blocks that gets cached during this time (by other scan) can occupy the L2 cache and so the cells that are added in the Result are now corrupted because the actual blocks in the L2 cache are overwritten. > HBASE-17917 closes the scanners while a scan is in progess for switching over > to stream reads > - > > Key: HBASE-18055 > URL: https://issues.apache.org/jira/browse/HBASE-18055 > Project: HBase > Issue Type: Bug > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > > In HBASE-17917 tries to switch from pread to stream read when a specific size > of bytes are read. So in order to switch over, it closes the existing > scanners and creates a new scanners with pread=false. > When we close the exisitng scanners - if the blocks are served from offheap > cache we will decrement the ref count on those blocks and if it becomes zero > we make the block ready for eviction. Then there is a chance that the result > could be corrupted if new blocks occupy the cache. So the expectation was > that till the RPC call completes the response we will hold on to the blocks > that are referred by the scan. (except the last one). So trying to switch > over to stream read will break this expectation and hence > TestBlockEvictionfromclient fails. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-18055) HBASE-17917 closes the scanners while a scan is in progess for switching over to stream reads
[ https://issues.apache.org/jira/browse/HBASE-18055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011847#comment-16011847 ] ramkrishna.s.vasudevan edited comment on HBASE-18055 at 5/16/17 6:55 AM: - Let me try to explain. For simplicity sake say we have one row per block. We have totally 4 rows (4 blocks). All are cached in L2. The scan is reading 3 blocks (3 rows). Tries with the next next() call. Sees that the preadMaxBytes is reached and creates new scanners and seeks to the 3rd row (since row 3 was latest). And then closes all the older scanners. So the block 1 and block 2 (of row 1 and row2) are now decremented and ref count is 0 since older scanners refers to these blocks. block 3 would have been incremented to 2 and then while closing it would have become 1. But since block 1 or block 2 are already decremented to 0, it means any other new blocks that gets cached during this time (by other scan) can occupy the L2 cache and so the cells that are added in the Result are now corrupted because the actual blocks in the L2 cache are overwritten. was (Author: ram_krish): Let me try to explain. For simplicity sake say we have one row per block. All are cached in L2. The scan is reading 3 blocks (3 rows). Tries with the next next() call. Sees that the preadMaxBytes is reached and creates new scanners and seeks to the 3rd row (since row 3 was latest). And then closes all the older scanners. So the block 1 and block 2 (of row 1 and row2) are now decremented and ref count is 0 since older scanners refers to these blocks. block 3 would have been incremented to 2 and then while closing it would have become 1. But since block 1 or block 2 are already decremented to 0, it means any other new blocks that gets cached during this time (by other scan) can occupy the L2 cache and so the cells that are added in the Result are now corrupted because the actual blocks in the L2 cache are overwritten. > HBASE-17917 closes the scanners while a scan is in progess for switching over > to stream reads > - > > Key: HBASE-18055 > URL: https://issues.apache.org/jira/browse/HBASE-18055 > Project: HBase > Issue Type: Bug > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > > In HBASE-17917 tries to switch from pread to stream read when a specific size > of bytes are read. So in order to switch over, it closes the existing > scanners and creates a new scanners with pread=false. > When we close the exisitng scanners - if the blocks are served from offheap > cache we will decrement the ref count on those blocks and if it becomes zero > we make the block ready for eviction. Then there is a chance that the result > could be corrupted if new blocks occupy the cache. So the expectation was > that till the RPC call completes the response we will hold on to the blocks > that are referred by the scan. (except the last one). So trying to switch > over to stream read will break this expectation and hence > TestBlockEvictionfromclient fails. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18055) HBASE-17917 closes the scanners while a scan is in progess for switching over to stream reads
[ https://issues.apache.org/jira/browse/HBASE-18055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011847#comment-16011847 ] ramkrishna.s.vasudevan commented on HBASE-18055: Let me try to explain. For simplicity sake say we have one row per block. All are cached in L2. The scan is reading 3 blocks (3 rows). Tries with the next next() call. Sees that the preadMaxBytes is reached and creates new scanners and seeks to the 3rd row (since row 3 was latest). And then closes all the older scanners. So the block 1 and block 2 (of row 1 and row2) are now decremented and ref count is 0 since older scanners refers to these blocks. block 3 would have been incremented to 2 and then while closing it would have become 1. But since block 1 or block 2 are already decremented to 0, it means any other new blocks that gets cached during this time (by other scan) can occupy the L2 cache and so the cells that are added in the Result are now corrupted because the actual blocks in the L2 cache are overwritten. > HBASE-17917 closes the scanners while a scan is in progess for switching over > to stream reads > - > > Key: HBASE-18055 > URL: https://issues.apache.org/jira/browse/HBASE-18055 > Project: HBase > Issue Type: Bug > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > > In HBASE-17917 tries to switch from pread to stream read when a specific size > of bytes are read. So in order to switch over, it closes the existing > scanners and creates a new scanners with pread=false. > When we close the exisitng scanners - if the blocks are served from offheap > cache we will decrement the ref count on those blocks and if it becomes zero > we make the block ready for eviction. Then there is a chance that the result > could be corrupted if new blocks occupy the cache. So the expectation was > that till the RPC call completes the response we will hold on to the blocks > that are referred by the scan. (except the last one). So trying to switch > over to stream read will break this expectation and hence > TestBlockEvictionfromclient fails. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18053) AsyncTableResultScanner will hang when scan wrong column family
[
https://issues.apache.org/jira/browse/HBASE-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011845#comment-16011845
]
Hadoop QA commented on HBASE-18053:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 26s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 55s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 45s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 100m 18s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
54s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 155m 19s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestMobStoreScanner |
| | hadoop.hbase.snapshot.TestMobExportSnapshot |
| | hadoop.hbase.snapshot.TestMobFlushSnapshotFromClient |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestFromClientSide |
| | org.apache.hadoop.hbase.client.TestMetaWithReplicas |
| | org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.10.1 Server=1.10.1 Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12868224/HBASE-18053-v1.patch |
| JIRA Issue | HBASE-18053 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 6913df53b524 3.19.0-25-generic #26~14.04.1-Ubuntu SMP Fri Jul
24 21:16:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 3765077 |
[jira] [Commented] (HBASE-18055) HBASE-17917 closes the scanners while a scan is in progess for switching over to stream reads
[ https://issues.apache.org/jira/browse/HBASE-18055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011832#comment-16011832 ] Duo Zhang commented on HBASE-18055: --- And in fact, we will open new scanners and seek them before closing the old scanners. I think they will just reference the same blocks? > HBASE-17917 closes the scanners while a scan is in progess for switching over > to stream reads > - > > Key: HBASE-18055 > URL: https://issues.apache.org/jira/browse/HBASE-18055 > Project: HBase > Issue Type: Bug > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > > In HBASE-17917 tries to switch from pread to stream read when a specific size > of bytes are read. So in order to switch over, it closes the existing > scanners and creates a new scanners with pread=false. > When we close the exisitng scanners - if the blocks are served from offheap > cache we will decrement the ref count on those blocks and if it becomes zero > we make the block ready for eviction. Then there is a chance that the result > could be corrupted if new blocks occupy the cache. So the expectation was > that till the RPC call completes the response we will hold on to the blocks > that are referred by the scan. (except the last one). So trying to switch > over to stream read will break this expectation and hence > TestBlockEvictionfromclient fails. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18055) HBASE-17917 closes the scanners while a scan is in progess for switching over to stream reads
[ https://issues.apache.org/jira/browse/HBASE-18055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011824#comment-16011824 ] Duo Zhang commented on HBASE-18055: --- Pardon me, I do not understand what is going on here. You mean that the newly created scanner will reference to a block whose ref count is 0 so we may get incorrect result? > HBASE-17917 closes the scanners while a scan is in progess for switching over > to stream reads > - > > Key: HBASE-18055 > URL: https://issues.apache.org/jira/browse/HBASE-18055 > Project: HBase > Issue Type: Bug > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > > In HBASE-17917 tries to switch from pread to stream read when a specific size > of bytes are read. So in order to switch over, it closes the existing > scanners and creates a new scanners with pread=false. > When we close the exisitng scanners - if the blocks are served from offheap > cache we will decrement the ref count on those blocks and if it becomes zero > we make the block ready for eviction. Then there is a chance that the result > could be corrupted if new blocks occupy the cache. So the expectation was > that till the RPC call completes the response we will hold on to the blocks > that are referred by the scan. (except the last one). So trying to switch > over to stream read will break this expectation and hence > TestBlockEvictionfromclient fails. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18055) HBASE-17917 closes the scanners while a scan is in progess for switching over to stream reads
[ https://issues.apache.org/jira/browse/HBASE-18055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011825#comment-16011825 ] Anoop Sam John commented on HBASE-18055: bq.creates a new scanners with pread=false Create new scanners means we will again look for getBlock and this time the block wont be avail in L2 cache right? Did not see the new code. Might be am missing some thing > HBASE-17917 closes the scanners while a scan is in progess for switching over > to stream reads > - > > Key: HBASE-18055 > URL: https://issues.apache.org/jira/browse/HBASE-18055 > Project: HBase > Issue Type: Bug > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > > In HBASE-17917 tries to switch from pread to stream read when a specific size > of bytes are read. So in order to switch over, it closes the existing > scanners and creates a new scanners with pread=false. > When we close the exisitng scanners - if the blocks are served from offheap > cache we will decrement the ref count on those blocks and if it becomes zero > we make the block ready for eviction. Then there is a chance that the result > could be corrupted if new blocks occupy the cache. So the expectation was > that till the RPC call completes the response we will hold on to the blocks > that are referred by the scan. (except the last one). So trying to switch > over to stream read will break this expectation and hence > TestBlockEvictionfromclient fails. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18051) balance_rsgroup still run when the Load Balancer is not enabled.
[
https://issues.apache.org/jira/browse/HBASE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011818#comment-16011818
]
Hadoop QA commented on HBASE-18051:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 30s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
60m 37s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 32s
{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 81m 4s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12868228/HBASE-18051-v3.patch |
| JIRA Issue | HBASE-18051 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 516ff5ee1e27 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3765077 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6805/testReport/ |
| modules | C: hbase-rsgroup U: hbase-rsgroup |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6805/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> balance_rsgroup still run when the Load Balancer is not enabled.
>
>
> Key: HBASE-18051
> URL: https://issues.apache.org/jira/browse/HBASE-18051
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
>Affects Versions: 2.0.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
> Fix For: 2.0.0
>
> Attach
[jira] [Commented] (HBASE-18055) HBASE-17917 closes the scanners while a scan is in progess for switching over to stream reads
[ https://issues.apache.org/jira/browse/HBASE-18055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011812#comment-16011812 ] ramkrishna.s.vasudevan commented on HBASE-18055: [~Apache9]. FYI. > HBASE-17917 closes the scanners while a scan is in progess for switching over > to stream reads > - > > Key: HBASE-18055 > URL: https://issues.apache.org/jira/browse/HBASE-18055 > Project: HBase > Issue Type: Bug > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > > In HBASE-17917 tries to switch from pread to stream read when a specific size > of bytes are read. So in order to switch over, it closes the existing > scanners and creates a new scanners with pread=false. > When we close the exisitng scanners - if the blocks are served from offheap > cache we will decrement the ref count on those blocks and if it becomes zero > we make the block ready for eviction. Then there is a chance that the result > could be corrupted if new blocks occupy the cache. So the expectation was > that till the RPC call completes the response we will hold on to the blocks > that are referred by the scan. (except the last one). So trying to switch > over to stream read will break this expectation and hence > TestBlockEvictionfromclient fails. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-18055) HBASE-17917 closes the scanners while a scan is in progess for switching over to stream reads
ramkrishna.s.vasudevan created HBASE-18055: -- Summary: HBASE-17917 closes the scanners while a scan is in progess for switching over to stream reads Key: HBASE-18055 URL: https://issues.apache.org/jira/browse/HBASE-18055 Project: HBase Issue Type: Bug Components: regionserver, Scanners Affects Versions: 2.0.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 In HBASE-17917 tries to switch from pread to stream read when a specific size of bytes are read. So in order to switch over, it closes the existing scanners and creates a new scanners with pread=false. When we close the exisitng scanners - if the blocks are served from offheap cache we will decrement the ref count on those blocks and if it becomes zero we make the block ready for eviction. Then there is a chance that the result could be corrupted if new blocks occupy the cache. So the expectation was that till the RPC call completes the response we will hold on to the blocks that are referred by the scan. (except the last one). So trying to switch over to stream read will break this expectation and hence TestBlockEvictionfromclient fails. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-11013) Clone Snapshots on Secure Cluster Should provide option to apply Retained User Permissions
[
https://issues.apache.org/jira/browse/HBASE-11013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011808#comment-16011808
]
Hadoop QA commented on HBASE-11013:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 53s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 1s
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 1s
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 17 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 15s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
10s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 29s
{color} | {color:green} branch-1 passed with JDK v1.8.0_131 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 38s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 9m
41s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
10s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 20s
{color} | {color:red} hbase-server in branch-1 has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} branch-1 passed with JDK v1.8.0_131 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 30s
{color} | {color:green} the patch passed with JDK v1.8.0_131 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 41s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 41s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 42s {color}
| {color:red} hbase-server-jdk1.7.0_80 with JDK v1.7.0_80 generated 1 new + 4
unchanged - 1 fixed = 5 total (was 5) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 9m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
17m 41s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s
{color} | {color:green} the patch passed with JDK v1.8.0_131 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 7s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color}
[jira] [Commented] (HBASE-18038) Rename StoreFile to HStoreFile and add a StoreFile interface for CP
[ https://issues.apache.org/jira/browse/HBASE-18038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011785#comment-16011785 ] Duo Zhang commented on HBASE-18038: --- Let me check the failed UTs. > Rename StoreFile to HStoreFile and add a StoreFile interface for CP > --- > > Key: HBASE-18038 > URL: https://issues.apache.org/jira/browse/HBASE-18038 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, regionserver >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18038.patch, HBASE-18038-v1.patch, > HBASE-18038-v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-11013) Clone Snapshots on Secure Cluster Should provide option to apply Retained User Permissions
[
https://issues.apache.org/jira/browse/HBASE-11013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011782#comment-16011782
]
Hadoop QA commented on HBASE-11013:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m 26s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 0s
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 0s
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 44s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 40s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 16m
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
57s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 55s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 16m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 58s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 28s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 29s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 110m 39s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 8s
{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m 17s
{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
0s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 222m 23s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hado
[jira] [Commented] (HBASE-18049) It is not necessary to re-open the region when MOB files cannot be found
[
https://issues.apache.org/jira/browse/HBASE-18049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011778#comment-16011778
]
Anoop Sam John commented on HBASE-18049:
bq.Right, this is a simple solution which can probably fix the issue. But I
think users need to investigate the cases a little before running such
commands. Given such info in logs might mislead users to directly run the
compaction? I think we'd better not add this for now?
Ok. Just asked as recently some users faced this.
> It is not necessary to re-open the region when MOB files cannot be found
>
>
> Key: HBASE-18049
> URL: https://issues.apache.org/jira/browse/HBASE-18049
> Project: HBase
> Issue Type: Bug
> Components: Scanners
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Fix For: 2.0.0
>
> Attachments: HBASE-18049.patch
>
>
> In HBASE-17712, we try to re-open the region when store files cannot be
> found. This is useful for store files in a region, but is not necessary when
> the MOB files cannot be found, because the store files in a region only
> contain the references to the MOB files and a re-open of a region doesn't
> help the lost MOB files.
> In this JIRA, we will directly throw DNRIOE only when the MOB files are not
> found in {{MobStoreScanner}} and {{ReversedMobStoreScanner}}. Other logics
> keep the same.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18038) Rename StoreFile to HStoreFile and add a StoreFile interface for CP
[
https://issues.apache.org/jira/browse/HBASE-18038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011774#comment-16011774
]
Hadoop QA commented on HBASE-18038:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 30s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 38 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 27s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
60m 0s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 162m 13s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
54s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 252m 56s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestRegionReplicaFailover |
| | hadoop.hbase.security.access.TestAccessController |
| | hadoop.hbase.mob.compactions.TestMobCompactor |
| | hadoop.hbase.security.token.TestZKSecretWatcher |
| | hadoop.hbase.regionserver.TestMobStoreCompaction |
| | hadoop.hbase.regionserver.TestEncryptionKeyRotation |
| | hadoop.hbase.regionserver.TestPerColumnFamilyFlush |
| | hadoop.hbase.mob.compactions.TestPartitionedMobCompactor |
| | hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster |
| | hadoop.hbase.regionserver.TestCorruptedRegionStoreFile |
| | hadoop.hbase.regionserver.TestReversibleScanners |
| Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestMobSecureExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestExportSnapshot |
| | org.apache.hadoop.hbase.TestIOFencing |
| | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12868202/HBASE-18038-v1.patch |
| JIRA Issue | HBASE-18038 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux ee9db86a07eb 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC
[jira] [Commented] (HBASE-18049) It is not necessary to re-open the region when MOB files cannot be found
[
https://issues.apache.org/jira/browse/HBASE-18049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011767#comment-16011767
]
Jingcheng Du commented on HBASE-18049:
--
Thanks [~yuzhih...@gmail.com] and [~anoop.hbase] for the comments!
bq. So here in new change as well, we need getCause() check?
Right, we should add this.
bq. A solution to restore the state and data is to do a MOB major compaction
right?
Right, this is a simple solution which can probably fix the issue. But I think
users need to investigate the cases a little before running such commands.
Given such info in logs might mislead users to directly run the compaction? I
think we'd better not add this for now?
> It is not necessary to re-open the region when MOB files cannot be found
>
>
> Key: HBASE-18049
> URL: https://issues.apache.org/jira/browse/HBASE-18049
> Project: HBase
> Issue Type: Bug
> Components: Scanners
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Fix For: 2.0.0
>
> Attachments: HBASE-18049.patch
>
>
> In HBASE-17712, we try to re-open the region when store files cannot be
> found. This is useful for store files in a region, but is not necessary when
> the MOB files cannot be found, because the store files in a region only
> contain the references to the MOB files and a re-open of a region doesn't
> help the lost MOB files.
> In this JIRA, we will directly throw DNRIOE only when the MOB files are not
> found in {{MobStoreScanner}} and {{ReversedMobStoreScanner}}. Other logics
> keep the same.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18040) TestBlockEvictionFromClient fails in master branch
[
https://issues.apache.org/jira/browse/HBASE-18040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011766#comment-16011766
]
ramkrishna.s.vasudevan commented on HBASE-18040:
Found the issue. Will raise a new JIRA to fix it. It needs some discussion.
Will link this failure to the other new JIRA.
> TestBlockEvictionFromClient fails in master branch
> --
>
> Key: HBASE-18040
> URL: https://issues.apache.org/jira/browse/HBASE-18040
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: ramkrishna.s.vasudevan
>
> According to
> https://builds.apache.org/job/HBASE-Flaky-Tests/16000/#showFailuresLink , the
> test first failed with commit 0ae0edcd630aa1dcb6c47ea11fa4367ae0a5baa8
> On Linux, I got the following failures:
> {code}
> testParallelGetsAndScans(org.apache.hadoop.hbase.client.TestBlockEvictionFromClient)
> Time elapsed: 3.016 sec <<< FAILURE!
> java.lang.AssertionError: expected:<3> but was:<6>
> at
> org.apache.hadoop.hbase.client.TestBlockEvictionFromClient.checkForBlockEviction(TestBlockEvictionFromClient.java:1308)
> at
> org.apache.hadoop.hbase.client.TestBlockEvictionFromClient.testParallelGetsAndScans(TestBlockEvictionFromClient.java:293)
> testBlockRefCountAfterSplits(org.apache.hadoop.hbase.client.TestBlockEvictionFromClient)
> Time elapsed: 7.786 sec <<< FAILURE!
> java.lang.AssertionError: expected:<0> but was:<1>
> at
> org.apache.hadoop.hbase.client.TestBlockEvictionFromClient.iterateBlockCache(TestBlockEvictionFromClient.java:1215)
> at
> org.apache.hadoop.hbase.client.TestBlockEvictionFromClient.testBlockRefCountAfterSplits(TestBlockEvictionFromClient.java:607)
> testParallelGetsAndScanWithWrappedRegionScanner(org.apache.hadoop.hbase.client.TestBlockEvictionFromClient)
> Time elapsed: 2.631 sec <<< FAILURE!
> java.lang.AssertionError: expected:<3> but was:<6>
> at
> org.apache.hadoop.hbase.client.TestBlockEvictionFromClient.checkForBlockEviction(TestBlockEvictionFromClient.java:1322)
> at
> org.apache.hadoop.hbase.client.TestBlockEvictionFromClient.testParallelGetsAndScanWithWrappedRegionScanner(TestBlockEvictionFromClient.java:839)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18053) AsyncTableResultScanner will hang when scan wrong column family
[
https://issues.apache.org/jira/browse/HBASE-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011760#comment-16011760
]
Hadoop QA commented on HBASE-18053:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 37s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 49s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 41m 25s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m 12s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.10.1 Server=1.10.1 Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12868221/HBASE-18053.patch |
| JIRA Issue | HBASE-18053 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux c59267d594b2 3.19.0-25-generic #26~14.04.1-Ubuntu SMP Fri Jul
24 21:16:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3765077 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6801/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/6801/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Commented] (HBASE-18026) ProtobufUtil seems to do extra array copying
[ https://issues.apache.org/jira/browse/HBASE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011757#comment-16011757 ] Hudson commented on HBASE-18026: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #178 (See [https://builds.apache.org/job/HBase-1.3-JDK8/178/]) Amend HBASE-18026 ProtobufUtil seems to do extra array copying (apurtell: rev d18eb62d8d0f02ea822090f409c2ded6f5279c3d) * (edit) hbase-protocol/src/main/java/com/google/protobuf/HBaseZeroCopyByteString.java > ProtobufUtil seems to do extra array copying > > > Key: HBASE-18026 > URL: https://issues.apache.org/jira/browse/HBASE-18026 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-18026.addendum.patch, > HBASE-18026.branch-1.v1.patch, HBASE-18026.master.v1.patch > > > In ProtobufUtil, the protobuf fields are copied into an array using > toByteArray(). These are then passed into the KeyValue constructor which > does another copy. > It seems like we can avoid a copy here by using > HBaseZeroCopyByteString#zeroCopyGetBytes() ? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18051) balance_rsgroup still run when the Load Balancer is not enabled.
[ https://issues.apache.org/jira/browse/HBASE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated HBASE-18051: -- Attachment: HBASE-18051-v3.patch > balance_rsgroup still run when the Load Balancer is not enabled. > > > Key: HBASE-18051 > URL: https://issues.apache.org/jira/browse/HBASE-18051 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Fix For: 2.0.0 > > Attachments: HBASE-18051-v1.patch, HBASE-18051-v2.patch, > HBASE-18051-v3.patch > > > When the Load Balancer is not enabled, the command "balancer" will not be > executed , but the command "balance_rsgroup" can still run, which is not > reasonable.So we need to fix it -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18051) balance_rsgroup still run when the Load Balancer is not enabled.
[ https://issues.apache.org/jira/browse/HBASE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011747#comment-16011747 ] Guangxu Cheng commented on HBASE-18051: --- v3: Add a unit test and fix the failed tests. > balance_rsgroup still run when the Load Balancer is not enabled. > > > Key: HBASE-18051 > URL: https://issues.apache.org/jira/browse/HBASE-18051 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Fix For: 2.0.0 > > Attachments: HBASE-18051-v1.patch, HBASE-18051-v2.patch, > HBASE-18051-v3.patch > > > When the Load Balancer is not enabled, the command "balancer" will not be > executed , but the command "balance_rsgroup" can still run, which is not > reasonable.So we need to fix it -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18027) Replication should respect RPC size limits when batching edits
[
https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011745#comment-16011745
]
Hadoop QA commented on HBASE-18027:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 31m 44s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
58m 28s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 52s
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 176m 44s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
35s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 293m 53s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Inconsistent synchronization of
org.apache.hadoop.hbase.replication.regionserver.ReplicationSinkManager.sinks;
locked 72% of time Unsynchronized access at ReplicationSinkManager.java:72% of
time Unsynchronized access at ReplicationSinkManager.java:[line 175] |
| Failed junit tests | hadoop.hbase.client.TestAsyncBalancerAdminApi |
| | hadoop.hbase.client.TestMobSnapshotCloneIndependence |
| | hadoop.hbase.client.TestAsyncTableAdminApi |
| | hadoop.hbase.security.access.TestCoprocessorWhitelistMasterObserver |
| Timed out junit tests |
org.apache.hadoop.hbase.replication.regionserver.TestWALEntryStream |
| | org.apache.hadoop.hbase.client.TestFromClientSide3 |
| | org.apache.hadoop.hbase.snapshot.TestExportSnapshot |
| | org.apache.hadoop.hbase.client.TestHCM |
| | org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClient |
| |
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas |
| | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12868182/HBASE-18027.patch |
| JIRA Issue | HBASE-18027 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 565cb4f8f76d 4.8.3-std-
[jira] [Commented] (HBASE-18026) ProtobufUtil seems to do extra array copying
[ https://issues.apache.org/jira/browse/HBASE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011739#comment-16011739 ] Hudson commented on HBASE-18026: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK7 #164 (See [https://builds.apache.org/job/HBase-1.3-JDK7/164/]) Amend HBASE-18026 ProtobufUtil seems to do extra array copying (apurtell: rev d18eb62d8d0f02ea822090f409c2ded6f5279c3d) * (edit) hbase-protocol/src/main/java/com/google/protobuf/HBaseZeroCopyByteString.java > ProtobufUtil seems to do extra array copying > > > Key: HBASE-18026 > URL: https://issues.apache.org/jira/browse/HBASE-18026 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-18026.addendum.patch, > HBASE-18026.branch-1.v1.patch, HBASE-18026.master.v1.patch > > > In ProtobufUtil, the protobuf fields are copied into an array using > toByteArray(). These are then passed into the KeyValue constructor which > does another copy. > It seems like we can avoid a copy here by using > HBaseZeroCopyByteString#zeroCopyGetBytes() ? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18035) Meta replica does not give any primaryOperationTimeout to primary meta region
[
https://issues.apache.org/jira/browse/HBASE-18035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011737#comment-16011737
]
Hadoop QA commented on HBASE-18035:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m 52s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 47s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 22s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
32m 32s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 56s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 30s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 123m 54s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
0s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 199m 51s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestMobStoreScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12868201/HBASE-18035-master-v001.patch
|
| JIRA Issue | HBASE-18035 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux af9a95d24e3e 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3765077 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6798/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit t
[jira] [Commented] (HBASE-18026) ProtobufUtil seems to do extra array copying
[ https://issues.apache.org/jira/browse/HBASE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011732#comment-16011732 ] Hudson commented on HBASE-18026: SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #135 (See [https://builds.apache.org/job/HBase-1.2-JDK7/135/]) Amend HBASE-18026 ProtobufUtil seems to do extra array copying (apurtell: rev 719ab32cc6910cf393ce235315190ffd8dd7a3c8) * (edit) hbase-protocol/src/main/java/com/google/protobuf/HBaseZeroCopyByteString.java > ProtobufUtil seems to do extra array copying > > > Key: HBASE-18026 > URL: https://issues.apache.org/jira/browse/HBASE-18026 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-18026.addendum.patch, > HBASE-18026.branch-1.v1.patch, HBASE-18026.master.v1.patch > > > In ProtobufUtil, the protobuf fields are copied into an array using > toByteArray(). These are then passed into the KeyValue constructor which > does another copy. > It seems like we can avoid a copy here by using > HBaseZeroCopyByteString#zeroCopyGetBytes() ? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18051) balance_rsgroup still run when the Load Balancer is not enabled.
[
https://issues.apache.org/jira/browse/HBASE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011729#comment-16011729
]
Hadoop QA commented on HBASE-18051:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 40s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 19s {color}
| {color:red} hbase-rsgroup in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 25s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.rsgroup.TestRSGroups |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12868215/HBASE-18051-v2.patch |
| JIRA Issue | HBASE-18051 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux d7190214c3fa 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 3765077 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6800/artifact/patchprocess/patch-unit-hbase-rsgroup.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/6800/artifact/patchprocess/patch-unit-hbase-rsgroup.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6800/testReport/ |
| modules | C: hbase-rsgroup U: hbase-rsgroup |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6800/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> balance_rsgro
[jira] [Updated] (HBASE-11013) Clone Snapshots on Secure Cluster Should provide option to apply Retained User Permissions
[
https://issues.apache.org/jira/browse/HBASE-11013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-11013:
-
Attachment: HBASE-11013.branch-1.v1.patch
> Clone Snapshots on Secure Cluster Should provide option to apply Retained
> User Permissions
> --
>
> Key: HBASE-11013
> URL: https://issues.apache.org/jira/browse/HBASE-11013
> Project: HBase
> Issue Type: Improvement
> Components: snapshots
>Reporter: Ted Yu
>Assignee: Zheng Hu
> Fix For: 2.0.0
>
> Attachments: HBASE-11013.branch-1.v1.patch,
> HBASE-11013.branch-1.v1.patch, HBASE-11013.master.addendum.patch,
> HBASE-11013.v1.patch, HBASE-11013.v2.patch, HBASE-11013.v3.patch,
> HBASE-11013.v3.patch, HBASE-11013.v4.patch, HBASE-11013.v4.patch
>
>
> Currently,
> {code}
> sudo su - test_user
> create 't1', 'f1'
> sudo su - hbase
> snapshot 't1', 'snap_one'
> clone_snapshot 'snap_one', 't2'
> {code}
> In this scenario the user - test_user would not have permissions for the
> clone table t2.
> We need to add improvement feature such that the permissions of the original
> table are recorded in snapshot metadata and an option is provided for
> applying them to the new table as part of the clone process.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18053) AsyncTableResultScanner will hang when scan wrong column family
[ https://issues.apache.org/jira/browse/HBASE-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011724#comment-16011724 ] Ted Yu commented on HBASE-18053: lgtm, pending QA > AsyncTableResultScanner will hang when scan wrong column family > --- > > Key: HBASE-18053 > URL: https://issues.apache.org/jira/browse/HBASE-18053 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18053.patch, HBASE-18053-v1.patch > > > AsynTableResultScanner did not call notify(). So the next will hang on > wait(). It is easy to fix. And will add a ut for this. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-11013) Clone Snapshots on Secure Cluster Should provide option to apply Retained User Permissions
[
https://issues.apache.org/jira/browse/HBASE-11013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011725#comment-16011725
]
Zheng Hu commented on HBASE-11013:
--
[~tedyu], The main differences is: branch-1 has no hbase-protocol-shaded
sub-module, so we need not to move AccessControl.proto to
hbase-protocol-shaded directory, it saves lots of generated code.
> Clone Snapshots on Secure Cluster Should provide option to apply Retained
> User Permissions
> --
>
> Key: HBASE-11013
> URL: https://issues.apache.org/jira/browse/HBASE-11013
> Project: HBase
> Issue Type: Improvement
> Components: snapshots
>Reporter: Ted Yu
>Assignee: Zheng Hu
> Fix For: 2.0.0
>
> Attachments: HBASE-11013.branch-1.v1.patch,
> HBASE-11013.master.addendum.patch, HBASE-11013.v1.patch,
> HBASE-11013.v2.patch, HBASE-11013.v3.patch, HBASE-11013.v3.patch,
> HBASE-11013.v4.patch, HBASE-11013.v4.patch
>
>
> Currently,
> {code}
> sudo su - test_user
> create 't1', 'f1'
> sudo su - hbase
> snapshot 't1', 'snap_one'
> clone_snapshot 'snap_one', 't2'
> {code}
> In this scenario the user - test_user would not have permissions for the
> clone table t2.
> We need to add improvement feature such that the permissions of the original
> table are recorded in snapshot metadata and an option is provided for
> applying them to the new table as part of the clone process.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18026) ProtobufUtil seems to do extra array copying
[ https://issues.apache.org/jira/browse/HBASE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011722#comment-16011722 ] Hudson commented on HBASE-18026: SUCCESS: Integrated in Jenkins build HBase-1.1-JDK8 #1953 (See [https://builds.apache.org/job/HBase-1.1-JDK8/1953/]) Amend HBASE-18026 ProtobufUtil seems to do extra array copying (apurtell: rev 2cff94c40933ce4f431f733b2de6d44e3e19c6f1) * (edit) hbase-protocol/src/main/java/com/google/protobuf/HBaseZeroCopyByteString.java > ProtobufUtil seems to do extra array copying > > > Key: HBASE-18026 > URL: https://issues.apache.org/jira/browse/HBASE-18026 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-18026.addendum.patch, > HBASE-18026.branch-1.v1.patch, HBASE-18026.master.v1.patch > > > In ProtobufUtil, the protobuf fields are copied into an array using > toByteArray(). These are then passed into the KeyValue constructor which > does another copy. > It seems like we can avoid a copy here by using > HBaseZeroCopyByteString#zeroCopyGetBytes() ? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18053) AsyncTableResultScanner will hang when scan wrong column family
[ https://issues.apache.org/jira/browse/HBASE-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011721#comment-16011721 ] Guanghao Zhang commented on HBASE-18053: I add the ut in AbstractTestAsyncTableScan. And TestAsyncTableScanAll extends AbstractTestAsyncTableScan, too. The scanAll method return a CompletableFuture. So the exception is a ExecutionExceotion and caused by NoSuchColumnFamilyException. So I changed the assert condition in v1 patch. Thanks. > AsyncTableResultScanner will hang when scan wrong column family > --- > > Key: HBASE-18053 > URL: https://issues.apache.org/jira/browse/HBASE-18053 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18053.patch, HBASE-18053-v1.patch > > > AsynTableResultScanner did not call notify(). So the next will hang on > wait(). It is easy to fix. And will add a ut for this. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-11013) Clone Snapshots on Secure Cluster Should provide option to apply Retained User Permissions
[
https://issues.apache.org/jira/browse/HBASE-11013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011719#comment-16011719
]
Ted Yu commented on HBASE-11013:
Add test category:
{code}
+public class TestSnapshotWithAcl extends SecureTestUtil {
{code}
> Clone Snapshots on Secure Cluster Should provide option to apply Retained
> User Permissions
> --
>
> Key: HBASE-11013
> URL: https://issues.apache.org/jira/browse/HBASE-11013
> Project: HBase
> Issue Type: Improvement
> Components: snapshots
>Reporter: Ted Yu
>Assignee: Zheng Hu
> Fix For: 2.0.0
>
> Attachments: HBASE-11013.branch-1.v1.patch,
> HBASE-11013.master.addendum.patch, HBASE-11013.v1.patch,
> HBASE-11013.v2.patch, HBASE-11013.v3.patch, HBASE-11013.v3.patch,
> HBASE-11013.v4.patch, HBASE-11013.v4.patch
>
>
> Currently,
> {code}
> sudo su - test_user
> create 't1', 'f1'
> sudo su - hbase
> snapshot 't1', 'snap_one'
> clone_snapshot 'snap_one', 't2'
> {code}
> In this scenario the user - test_user would not have permissions for the
> clone table t2.
> We need to add improvement feature such that the permissions of the original
> table are recorded in snapshot metadata and an option is provided for
> applying them to the new table as part of the clone process.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-11013) Clone Snapshots on Secure Cluster Should provide option to apply Retained User Permissions
[
https://issues.apache.org/jira/browse/HBASE-11013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011718#comment-16011718
]
Ted Yu commented on HBASE-11013:
Zheng:
Can you highlight anything unique to the branch-1 patch (compared to the master
branch patch) ?
> Clone Snapshots on Secure Cluster Should provide option to apply Retained
> User Permissions
> --
>
> Key: HBASE-11013
> URL: https://issues.apache.org/jira/browse/HBASE-11013
> Project: HBase
> Issue Type: Improvement
> Components: snapshots
>Reporter: Ted Yu
>Assignee: Zheng Hu
> Fix For: 2.0.0
>
> Attachments: HBASE-11013.branch-1.v1.patch,
> HBASE-11013.master.addendum.patch, HBASE-11013.v1.patch,
> HBASE-11013.v2.patch, HBASE-11013.v3.patch, HBASE-11013.v3.patch,
> HBASE-11013.v4.patch, HBASE-11013.v4.patch
>
>
> Currently,
> {code}
> sudo su - test_user
> create 't1', 'f1'
> sudo su - hbase
> snapshot 't1', 'snap_one'
> clone_snapshot 'snap_one', 't2'
> {code}
> In this scenario the user - test_user would not have permissions for the
> clone table t2.
> We need to add improvement feature such that the permissions of the original
> table are recorded in snapshot metadata and an option is provided for
> applying them to the new table as part of the clone process.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18053) AsyncTableResultScanner will hang when scan wrong column family
[ https://issues.apache.org/jira/browse/HBASE-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-18053: --- Description: AsynTableResultScanner did not call notify(). So the next will hang on wait(). It is easy to fix. And will add a ut for this. > AsyncTableResultScanner will hang when scan wrong column family > --- > > Key: HBASE-18053 > URL: https://issues.apache.org/jira/browse/HBASE-18053 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18053.patch, HBASE-18053-v1.patch > > > AsynTableResultScanner did not call notify(). So the next will hang on > wait(). It is easy to fix. And will add a ut for this. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17997) jruby-complete-1.6.8.jar is in cached_classpath.txt
[
https://issues.apache.org/jira/browse/HBASE-17997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011717#comment-16011717
]
Ted Yu commented on HBASE-17997:
[~waterlx]:
What do you think of the above ?
> jruby-complete-1.6.8.jar is in cached_classpath.txt
> ---
>
> Key: HBASE-17997
> URL: https://issues.apache.org/jira/browse/HBASE-17997
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>
> HBASE-15199 moves jruby-complete-1.6.8.jar to lib/ruby directory.
> However, jruby-complete-1.6.8.jar still appears in cached_classpath.txt
> This means that user would see exception similar to the following when
> starting hbase in standalone mode with s3a as rootdir :
> {code}
> 2017-05-04 16:41:32,854 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=38659]
> internal.S3MetadataResponseHandler: Unable to parse last modified date: Thu,
> 04 May 2017 16:27:09 GMT
> java.lang.IllegalStateException: Joda-time 2.2 or later version is required,
> but found version: null
> at com.amazonaws.util.DateUtils.handleException(DateUtils.java:149)
> at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:195)
> at
> com.amazonaws.services.s3.internal.ServiceUtils.parseRfc822Date(ServiceUtils.java:78)
> at
> com.amazonaws.services.s3.internal.AbstractS3ResponseHandler.populateObjectMetadata(AbstractS3ResponseHandler.java:115)
> at
> com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:52)
> at
> com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:30)
> at
> com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1072)
> at
> com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:746)
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
> at
> com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1191)
> at org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:148)
> at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:281)
> at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:364)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:75)
> at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.positionalReadWithExtra(HFileBlock.java:722)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1420)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1677)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1504)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:439)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:904)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:267)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:169)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:363)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:217)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2132)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2122)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5687)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2679)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2665)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2647)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6906)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6885)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2007)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-11013) Clone Snapshots on Secure Cluster Should provide option to apply Retained User Permissions
[
https://issues.apache.org/jira/browse/HBASE-11013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-11013:
-
Attachment: HBASE-11013.branch-1.v1.patch
> Clone Snapshots on Secure Cluster Should provide option to apply Retained
> User Permissions
> --
>
> Key: HBASE-11013
> URL: https://issues.apache.org/jira/browse/HBASE-11013
> Project: HBase
> Issue Type: Improvement
> Components: snapshots
>Reporter: Ted Yu
>Assignee: Zheng Hu
> Fix For: 2.0.0
>
> Attachments: HBASE-11013.branch-1.v1.patch,
> HBASE-11013.master.addendum.patch, HBASE-11013.v1.patch,
> HBASE-11013.v2.patch, HBASE-11013.v3.patch, HBASE-11013.v3.patch,
> HBASE-11013.v4.patch, HBASE-11013.v4.patch
>
>
> Currently,
> {code}
> sudo su - test_user
> create 't1', 'f1'
> sudo su - hbase
> snapshot 't1', 'snap_one'
> clone_snapshot 'snap_one', 't2'
> {code}
> In this scenario the user - test_user would not have permissions for the
> clone table t2.
> We need to add improvement feature such that the permissions of the original
> table are recorded in snapshot metadata and an option is provided for
> applying them to the new table as part of the clone process.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18053) AsyncTableResultScanner will hang when scan wrong column family
[ https://issues.apache.org/jira/browse/HBASE-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-18053: --- Attachment: HBASE-18053-v1.patch > AsyncTableResultScanner will hang when scan wrong column family > --- > > Key: HBASE-18053 > URL: https://issues.apache.org/jira/browse/HBASE-18053 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18053.patch, HBASE-18053-v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18026) ProtobufUtil seems to do extra array copying
[ https://issues.apache.org/jira/browse/HBASE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011702#comment-16011702 ] Hudson commented on HBASE-18026: SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #130 (See [https://builds.apache.org/job/HBase-1.2-JDK8/130/]) Amend HBASE-18026 ProtobufUtil seems to do extra array copying (apurtell: rev 719ab32cc6910cf393ce235315190ffd8dd7a3c8) * (edit) hbase-protocol/src/main/java/com/google/protobuf/HBaseZeroCopyByteString.java > ProtobufUtil seems to do extra array copying > > > Key: HBASE-18026 > URL: https://issues.apache.org/jira/browse/HBASE-18026 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-18026.addendum.patch, > HBASE-18026.branch-1.v1.patch, HBASE-18026.master.v1.patch > > > In ProtobufUtil, the protobuf fields are copied into an array using > toByteArray(). These are then passed into the KeyValue constructor which > does another copy. > It seems like we can avoid a copy here by using > HBaseZeroCopyByteString#zeroCopyGetBytes() ? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18053) AsyncTableResultScanner will hang when scan wrong column family
[ https://issues.apache.org/jira/browse/HBASE-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011697#comment-16011697 ] Ted Yu commented on HBASE-18053: You can use the following in the new test: @Test(expected = X.class) Please fill out description. > AsyncTableResultScanner will hang when scan wrong column family > --- > > Key: HBASE-18053 > URL: https://issues.apache.org/jira/browse/HBASE-18053 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18053.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18051) balance_rsgroup still run when the Load Balancer is not enabled.
[ https://issues.apache.org/jira/browse/HBASE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011693#comment-16011693 ] Guangxu Cheng commented on HBASE-18051: --- sure,I will do work for it > balance_rsgroup still run when the Load Balancer is not enabled. > > > Key: HBASE-18051 > URL: https://issues.apache.org/jira/browse/HBASE-18051 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Fix For: 2.0.0 > > Attachments: HBASE-18051-v1.patch, HBASE-18051-v2.patch > > > When the Load Balancer is not enabled, the command "balancer" will not be > executed , but the command "balance_rsgroup" can still run, which is not > reasonable.So we need to fix it -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18053) AsyncTableResultScanner will hang when scan wrong column family
[ https://issues.apache.org/jira/browse/HBASE-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-18053: --- Assignee: Guanghao Zhang Status: Patch Available (was: Open) > AsyncTableResultScanner will hang when scan wrong column family > --- > > Key: HBASE-18053 > URL: https://issues.apache.org/jira/browse/HBASE-18053 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18053.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18053) AsyncTableResultScanner will hang when scan wrong column family
[ https://issues.apache.org/jira/browse/HBASE-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-18053: --- Attachment: HBASE-18053.patch > AsyncTableResultScanner will hang when scan wrong column family > --- > > Key: HBASE-18053 > URL: https://issues.apache.org/jira/browse/HBASE-18053 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Guanghao Zhang > Attachments: HBASE-18053.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18026) ProtobufUtil seems to do extra array copying
[ https://issues.apache.org/jira/browse/HBASE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011689#comment-16011689 ] Hudson commented on HBASE-18026: SUCCESS: Integrated in Jenkins build HBase-1.1-JDK7 #1870 (See [https://builds.apache.org/job/HBase-1.1-JDK7/1870/]) Amend HBASE-18026 ProtobufUtil seems to do extra array copying (apurtell: rev 2cff94c40933ce4f431f733b2de6d44e3e19c6f1) * (edit) hbase-protocol/src/main/java/com/google/protobuf/HBaseZeroCopyByteString.java > ProtobufUtil seems to do extra array copying > > > Key: HBASE-18026 > URL: https://issues.apache.org/jira/browse/HBASE-18026 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-18026.addendum.patch, > HBASE-18026.branch-1.v1.patch, HBASE-18026.master.v1.patch > > > In ProtobufUtil, the protobuf fields are copied into an array using > toByteArray(). These are then passed into the KeyValue constructor which > does another copy. > It seems like we can avoid a copy here by using > HBaseZeroCopyByteString#zeroCopyGetBytes() ? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-18054) log when we add/remove failed servers in client
Sean Busbey created HBASE-18054:
---
Summary: log when we add/remove failed servers in client
Key: HBASE-18054
URL: https://issues.apache.org/jira/browse/HBASE-18054
Project: HBase
Issue Type: Bug
Components: Client, Operability
Affects Versions: 1.3.0, 1.2.0, 1.1.0, 2.0.0, 1.4.0
Reporter: Sean Busbey
Currently we log if a server is in the failed server list when we go to connect
to it, but we don't log anything about when the server got into the list.
This means we have to search the log for errors involving the same server name
that (hopefully) managed to get into the log within
{{FAILED_SERVER_EXPIRY_KEY}} milliseconds earlier (default 2 seconds).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18054) log when we add/remove failed servers in client
[
https://issues.apache.org/jira/browse/HBASE-18054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011690#comment-16011690
]
Sean Busbey commented on HBASE-18054:
-
log at DEBUG, I guess?
> log when we add/remove failed servers in client
> ---
>
> Key: HBASE-18054
> URL: https://issues.apache.org/jira/browse/HBASE-18054
> Project: HBase
> Issue Type: Bug
> Components: Client, Operability
>Affects Versions: 2.0.0, 1.1.0, 1.2.0, 1.3.0, 1.4.0
>Reporter: Sean Busbey
>
> Currently we log if a server is in the failed server list when we go to
> connect to it, but we don't log anything about when the server got into the
> list.
> This means we have to search the log for errors involving the same server
> name that (hopefully) managed to get into the log within
> {{FAILED_SERVER_EXPIRY_KEY}} milliseconds earlier (default 2 seconds).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18051) balance_rsgroup still run when the Load Balancer is not enabled.
[ https://issues.apache.org/jira/browse/HBASE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011688#comment-16011688 ] Ted Yu commented on HBASE-18051: Is it possible to add a unit test (to guard against regression) ? > balance_rsgroup still run when the Load Balancer is not enabled. > > > Key: HBASE-18051 > URL: https://issues.apache.org/jira/browse/HBASE-18051 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Fix For: 2.0.0 > > Attachments: HBASE-18051-v1.patch, HBASE-18051-v2.patch > > > When the Load Balancer is not enabled, the command "balancer" will not be > executed , but the command "balance_rsgroup" can still run, which is not > reasonable.So we need to fix it -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-18051) balance_rsgroup still run when the Load Balancer is not enabled.
[ https://issues.apache.org/jira/browse/HBASE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011682#comment-16011682 ] Guangxu Cheng edited comment on HBASE-18051 at 5/16/17 3:34 AM: v2 takes the suggestion from [~yuzhih...@gmail.com]. Thanks was (Author: andrewcheng): v2 takes the suggestion from [~yuzhih...@gmail.com] > balance_rsgroup still run when the Load Balancer is not enabled. > > > Key: HBASE-18051 > URL: https://issues.apache.org/jira/browse/HBASE-18051 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Fix For: 2.0.0 > > Attachments: HBASE-18051-v1.patch, HBASE-18051-v2.patch > > > When the Load Balancer is not enabled, the command "balancer" will not be > executed , but the command "balance_rsgroup" can still run, which is not > reasonable.So we need to fix it -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18051) balance_rsgroup still run when the Load Balancer is not enabled.
[ https://issues.apache.org/jira/browse/HBASE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated HBASE-18051: -- Attachment: HBASE-18051-v2.patch > balance_rsgroup still run when the Load Balancer is not enabled. > > > Key: HBASE-18051 > URL: https://issues.apache.org/jira/browse/HBASE-18051 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Fix For: 2.0.0 > > Attachments: HBASE-18051-v1.patch, HBASE-18051-v2.patch > > > When the Load Balancer is not enabled, the command "balancer" will not be > executed , but the command "balance_rsgroup" can still run, which is not > reasonable.So we need to fix it -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18051) balance_rsgroup still run when the Load Balancer is not enabled.
[ https://issues.apache.org/jira/browse/HBASE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011682#comment-16011682 ] Guangxu Cheng commented on HBASE-18051: --- v2 takes the suggestion from [~yuzhih...@gmail.com] > balance_rsgroup still run when the Load Balancer is not enabled. > > > Key: HBASE-18051 > URL: https://issues.apache.org/jira/browse/HBASE-18051 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Fix For: 2.0.0 > > Attachments: HBASE-18051-v1.patch, HBASE-18051-v2.patch > > > When the Load Balancer is not enabled, the command "balancer" will not be > executed , but the command "balance_rsgroup" can still run, which is not > reasonable.So we need to fix it -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18051) balance_rsgroup still run when the Load Balancer is not enabled.
[ https://issues.apache.org/jira/browse/HBASE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated HBASE-18051: -- Status: Patch Available (was: Open) > balance_rsgroup still run when the Load Balancer is not enabled. > > > Key: HBASE-18051 > URL: https://issues.apache.org/jira/browse/HBASE-18051 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Fix For: 2.0.0 > > Attachments: HBASE-18051-v1.patch, HBASE-18051-v2.patch > > > When the Load Balancer is not enabled, the command "balancer" will not be > executed , but the command "balance_rsgroup" can still run, which is not > reasonable.So we need to fix it -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-18053) AsyncTableResultScanner will hang when scan wrong column family
Guanghao Zhang created HBASE-18053: -- Summary: AsyncTableResultScanner will hang when scan wrong column family Key: HBASE-18053 URL: https://issues.apache.org/jira/browse/HBASE-18053 Project: HBase Issue Type: Bug Reporter: Guanghao Zhang -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18053) AsyncTableResultScanner will hang when scan wrong column family
[ https://issues.apache.org/jira/browse/HBASE-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-18053: --- Component/s: Client > AsyncTableResultScanner will hang when scan wrong column family > --- > > Key: HBASE-18053 > URL: https://issues.apache.org/jira/browse/HBASE-18053 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Guanghao Zhang > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18051) balance_rsgroup still run when the Load Balancer is not enabled.
[ https://issues.apache.org/jira/browse/HBASE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011662#comment-16011662 ] Ted Yu commented on HBASE-18051: If there is no other way, the above is fine by me. > balance_rsgroup still run when the Load Balancer is not enabled. > > > Key: HBASE-18051 > URL: https://issues.apache.org/jira/browse/HBASE-18051 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Fix For: 2.0.0 > > Attachments: HBASE-18051-v1.patch > > > When the Load Balancer is not enabled, the command "balancer" will not be > executed , but the command "balance_rsgroup" can still run, which is not > reasonable.So we need to fix it -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18051) balance_rsgroup still run when the Load Balancer is not enabled.
[ https://issues.apache.org/jira/browse/HBASE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011658#comment-16011658 ] Guangxu Cheng commented on HBASE-18051: --- bq.See if you can utilize existing method(s) and not add isBalancerOn() to MasterServices yep, isBalancerEnabled () calls the method isBalancerOn() in HMaster to get the state of the load balancer. In RSGroupAdminServer, the type of master is MasterServices, but isBalancerOn() is not defined in MasterServices, so I add isBalancerOn() to MasterServices. Can I use (HMaster) master).isBalancerOn() instead of adding isBalancerOn() to MasterServices? > balance_rsgroup still run when the Load Balancer is not enabled. > > > Key: HBASE-18051 > URL: https://issues.apache.org/jira/browse/HBASE-18051 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.0.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng > Fix For: 2.0.0 > > Attachments: HBASE-18051-v1.patch > > > When the Load Balancer is not enabled, the command "balancer" will not be > executed , but the command "balance_rsgroup" can still run, which is not > reasonable.So we need to fix it -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18026) ProtobufUtil seems to do extra array copying
[ https://issues.apache.org/jira/browse/HBASE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011654#comment-16011654 ] Hudson commented on HBASE-18026: SUCCESS: Integrated in Jenkins build HBase-1.2-IT #869 (See [https://builds.apache.org/job/HBase-1.2-IT/869/]) Amend HBASE-18026 ProtobufUtil seems to do extra array copying (apurtell: rev 719ab32cc6910cf393ce235315190ffd8dd7a3c8) * (edit) hbase-protocol/src/main/java/com/google/protobuf/HBaseZeroCopyByteString.java > ProtobufUtil seems to do extra array copying > > > Key: HBASE-18026 > URL: https://issues.apache.org/jira/browse/HBASE-18026 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-18026.addendum.patch, > HBASE-18026.branch-1.v1.patch, HBASE-18026.master.v1.patch > > > In ProtobufUtil, the protobuf fields are copied into an array using > toByteArray(). These are then passed into the KeyValue constructor which > does another copy. > It seems like we can avoid a copy here by using > HBaseZeroCopyByteString#zeroCopyGetBytes() ? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18026) ProtobufUtil seems to do extra array copying
[ https://issues.apache.org/jira/browse/HBASE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011645#comment-16011645 ] Hudson commented on HBASE-18026: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #46 (See [https://builds.apache.org/job/HBase-1.3-IT/46/]) Amend HBASE-18026 ProtobufUtil seems to do extra array copying (apurtell: rev d18eb62d8d0f02ea822090f409c2ded6f5279c3d) * (edit) hbase-protocol/src/main/java/com/google/protobuf/HBaseZeroCopyByteString.java > ProtobufUtil seems to do extra array copying > > > Key: HBASE-18026 > URL: https://issues.apache.org/jira/browse/HBASE-18026 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-18026.addendum.patch, > HBASE-18026.branch-1.v1.patch, HBASE-18026.master.v1.patch > > > In ProtobufUtil, the protobuf fields are copied into an array using > toByteArray(). These are then passed into the KeyValue constructor which > does another copy. > It seems like we can avoid a copy here by using > HBaseZeroCopyByteString#zeroCopyGetBytes() ? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14247) Separate the old WALs into different regionserver directories
[
https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011643#comment-16011643
]
Enis Soztutar commented on HBASE-14247:
---
Please feel free to go ahead.
> Separate the old WALs into different regionserver directories
> -
>
> Key: HBASE-14247
> URL: https://issues.apache.org/jira/browse/HBASE-14247
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-14247-v001.diff, HBASE-14247-v002.diff,
> HBASE-14247-v003.diff
>
>
> Currently all old WALs of regionservers are achieved into the single
> directory of oldWALs. In big clusters, because of long TTL of WAL or disabled
> replications, the number of files under oldWALs may reach the
> max-directory-items limit of HDFS, which will make the hbase cluster crashed.
> {quote}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
> limit=1048576 items=1048576
> {quote}
> A simple solution is to separate the old WALs into different directories
> according to the server name of the WAL.
> Suggestions are welcomed~ Thanks
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17359) Implement async admin
[ https://issues.apache.org/jira/browse/HBASE-17359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011642#comment-16011642 ] Zheng Hu commented on HBASE-17359: -- Assigned myself with HBASE-18003 and HBASE-18052. > Implement async admin > - > > Key: HBASE-17359 > URL: https://issues.apache.org/jira/browse/HBASE-17359 > Project: HBase > Issue Type: Umbrella > Components: Client >Reporter: Duo Zhang >Assignee: Guanghao Zhang > Labels: asynchronous > Fix For: 2.0.0 > > > And as we will return a CompletableFuture, I think we can just remove the > XXXAsync methods, and make all the methods blocking which means we will only > finish the CompletableFuture when the operation is done. User can choose > whether to wait on the returned CompletableFuture. > Convert this to a umbrella task. There maybe some sub-tasks. > 1. Table admin operations. > 2. Region admin operations. > 3. Namespace admin operations. > 4. Snapshot admin operations. > 5. Replication admin operations. > 6. Other operations, like quota, balance.. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-18052) Add doc, examples for async admin
[ https://issues.apache.org/jira/browse/HBASE-18052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu reassigned HBASE-18052: Assignee: Zheng Hu > Add doc, examples for async admin > - > > Key: HBASE-18052 > URL: https://issues.apache.org/jira/browse/HBASE-18052 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Guanghao Zhang >Assignee: Zheng Hu > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-18003) Fix flaky test TestAsyncTableAdminApi and TestAsyncSnapshotAdminApi
[ https://issues.apache.org/jira/browse/HBASE-18003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu reassigned HBASE-18003: Assignee: Zheng Hu > Fix flaky test TestAsyncTableAdminApi and TestAsyncSnapshotAdminApi > --- > > Key: HBASE-18003 > URL: https://issues.apache.org/jira/browse/HBASE-18003 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Guanghao Zhang >Assignee: Zheng Hu > Fix For: 2.0.0 > > > See > https://builds.apache.org/job/HBASE-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18026) ProtobufUtil seems to do extra array copying
[ https://issues.apache.org/jira/browse/HBASE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011638#comment-16011638 ] Hudson commented on HBASE-18026: FAILURE: Integrated in Jenkins build HBase-1.4 #735 (See [https://builds.apache.org/job/HBase-1.4/735/]) Amend HBASE-18026 ProtobufUtil seems to do extra array copying (apurtell: rev d8ef495063ebf794164603d0739fcb27dce106fa) * (edit) hbase-protocol/src/main/java/com/google/protobuf/HBaseZeroCopyByteString.java > ProtobufUtil seems to do extra array copying > > > Key: HBASE-18026 > URL: https://issues.apache.org/jira/browse/HBASE-18026 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-18026.addendum.patch, > HBASE-18026.branch-1.v1.patch, HBASE-18026.master.v1.patch > > > In ProtobufUtil, the protobuf fields are copied into an array using > toByteArray(). These are then passed into the KeyValue constructor which > does another copy. > It seems like we can avoid a copy here by using > HBaseZeroCopyByteString#zeroCopyGetBytes() ? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18043) Institute a hard limit for individual cell size that cannot be overridden by clients
[ https://issues.apache.org/jira/browse/HBASE-18043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011637#comment-16011637 ] Hudson commented on HBASE-18043: FAILURE: Integrated in Jenkins build HBase-1.4 #735 (See [https://builds.apache.org/job/HBase-1.4/735/]) HBASE-18043 Institute a hard limit for individual cell size that cannot (apurtell: rev 292226690ac842be4513366633f31bba4fa62d34) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * (edit) hbase-common/src/main/resources/hbase-default.xml * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java > Institute a hard limit for individual cell size that cannot be overridden by > clients > > > Key: HBASE-18043 > URL: https://issues.apache.org/jira/browse/HBASE-18043 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, regionserver >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-18043-branch-1.patch, HBASE-18043-branch-1.patch, > HBASE-18043-branch-1.patch, HBASE-18043-branch-1.patch, HBASE-18043.patch, > HBASE-18043.patch, HBASE-18043.patch, HBASE-18043.patch, HBASE-18043.patch, > HBASE-18043.patch > > > For sake of service protection we should not give absolute trust to clients > regarding resource limits that can impact stability, like cell size limits. > We should add a server side configuration that sets a hard limit for > individual cell size that cannot be overridden by the client. We can keep the > client side check, because it's expensive to reject a RPC that has already > come in. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18049) It is not necessary to re-open the region when MOB files cannot be found
[
https://issues.apache.org/jira/browse/HBASE-18049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011627#comment-16011627
]
Anoop Sam John commented on HBASE-18049:
A bug some where in the code makes this MOB data not found. The cell may be in
some other MOB file and we have an issue in the MOB ref cell read. A solution
to restore the state and data is to do a MOB major compaction right? May be we
should add that info the error log clearly.
> It is not necessary to re-open the region when MOB files cannot be found
>
>
> Key: HBASE-18049
> URL: https://issues.apache.org/jira/browse/HBASE-18049
> Project: HBase
> Issue Type: Bug
> Components: Scanners
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Fix For: 2.0.0
>
> Attachments: HBASE-18049.patch
>
>
> In HBASE-17712, we try to re-open the region when store files cannot be
> found. This is useful for store files in a region, but is not necessary when
> the MOB files cannot be found, because the store files in a region only
> contain the references to the MOB files and a re-open of a region doesn't
> help the lost MOB files.
> In this JIRA, we will directly throw DNRIOE only when the MOB files are not
> found in {{MobStoreScanner}} and {{ReversedMobStoreScanner}}. Other logics
> keep the same.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-11013) Clone Snapshots on Secure Cluster Should provide option to apply Retained User Permissions
[
https://issues.apache.org/jira/browse/HBASE-11013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-11013:
-
Attachment: HBASE-11013.v4.patch
Trigger QA again.
> Clone Snapshots on Secure Cluster Should provide option to apply Retained
> User Permissions
> --
>
> Key: HBASE-11013
> URL: https://issues.apache.org/jira/browse/HBASE-11013
> Project: HBase
> Issue Type: Improvement
> Components: snapshots
>Reporter: Ted Yu
>Assignee: Zheng Hu
> Fix For: 2.0.0
>
> Attachments: HBASE-11013.master.addendum.patch, HBASE-11013.v1.patch,
> HBASE-11013.v2.patch, HBASE-11013.v3.patch, HBASE-11013.v3.patch,
> HBASE-11013.v4.patch, HBASE-11013.v4.patch
>
>
> Currently,
> {code}
> sudo su - test_user
> create 't1', 'f1'
> sudo su - hbase
> snapshot 't1', 'snap_one'
> clone_snapshot 'snap_one', 't2'
> {code}
> In this scenario the user - test_user would not have permissions for the
> clone table t2.
> We need to add improvement feature such that the permissions of the original
> table are recorded in snapshot metadata and an option is provided for
> applying them to the new table as part of the clone process.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18049) It is not necessary to re-open the region when MOB files cannot be found
[
https://issues.apache.org/jira/browse/HBASE-18049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011621#comment-16011621
]
Anoop Sam John commented on HBASE-18049:
In above lines we have
{code}
if ((e instanceof FileNotFoundException) ||
(e.getCause() instanceof FileNotFoundException)) {
{code}
So here in new change as well, we need getCause() check? Just asking. I did
not check the flow to see how the Exception stack will be possibly be. Else
looks ok.
> It is not necessary to re-open the region when MOB files cannot be found
>
>
> Key: HBASE-18049
> URL: https://issues.apache.org/jira/browse/HBASE-18049
> Project: HBase
> Issue Type: Bug
> Components: Scanners
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Fix For: 2.0.0
>
> Attachments: HBASE-18049.patch
>
>
> In HBASE-17712, we try to re-open the region when store files cannot be
> found. This is useful for store files in a region, but is not necessary when
> the MOB files cannot be found, because the store files in a region only
> contain the references to the MOB files and a re-open of a region doesn't
> help the lost MOB files.
> In this JIRA, we will directly throw DNRIOE only when the MOB files are not
> found in {{MobStoreScanner}} and {{ReversedMobStoreScanner}}. Other logics
> keep the same.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-14247) Separate the old WALs into different regionserver directories
[
https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011613#comment-16011613
]
Guanghao Zhang commented on HBASE-14247:
[~enis] Do you work on this issue? I have a patch for our 0.98 branch. If you
not work on this, I can help to prepare a patch for master branch. Thanks.
> Separate the old WALs into different regionserver directories
> -
>
> Key: HBASE-14247
> URL: https://issues.apache.org/jira/browse/HBASE-14247
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-14247-v001.diff, HBASE-14247-v002.diff,
> HBASE-14247-v003.diff
>
>
> Currently all old WALs of regionservers are achieved into the single
> directory of oldWALs. In big clusters, because of long TTL of WAL or disabled
> replications, the number of files under oldWALs may reach the
> max-directory-items limit of HDFS, which will make the hbase cluster crashed.
> {quote}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
> limit=1048576 items=1048576
> {quote}
> A simple solution is to separate the old WALs into different directories
> according to the server name of the WAL.
> Suggestions are welcomed~ Thanks
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17994) Add async client test to Performance Evaluation tool
[ https://issues.apache.org/jira/browse/HBASE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17994: --- Issue Type: Sub-task (was: New Feature) Parent: HBASE-16833 > Add async client test to Performance Evaluation tool > > > Key: HBASE-17994 > URL: https://issues.apache.org/jira/browse/HBASE-17994 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17359) Implement async admin
[ https://issues.apache.org/jira/browse/HBASE-17359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011605#comment-16011605 ] Guanghao Zhang commented on HBASE-17359: There are two sub-task need to resolve. HBASE-18003 is about the flaky ut. And HBASE-18052 is about doc, examples for async admin. Thanks. > Implement async admin > - > > Key: HBASE-17359 > URL: https://issues.apache.org/jira/browse/HBASE-17359 > Project: HBase > Issue Type: Umbrella > Components: Client >Reporter: Duo Zhang >Assignee: Guanghao Zhang > Labels: asynchronous > Fix For: 2.0.0 > > > And as we will return a CompletableFuture, I think we can just remove the > XXXAsync methods, and make all the methods blocking which means we will only > finish the CompletableFuture when the operation is done. User can choose > whether to wait on the returned CompletableFuture. > Convert this to a umbrella task. There maybe some sub-tasks. > 1. Table admin operations. > 2. Region admin operations. > 3. Namespace admin operations. > 4. Snapshot admin operations. > 5. Replication admin operations. > 6. Other operations, like quota, balance.. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-18052) Add doc, examples for async admin
Guanghao Zhang created HBASE-18052: -- Summary: Add doc, examples for async admin Key: HBASE-18052 URL: https://issues.apache.org/jira/browse/HBASE-18052 Project: HBase Issue Type: Sub-task Reporter: Guanghao Zhang -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18043) Institute a hard limit for individual cell size that cannot be overridden by clients
[ https://issues.apache.org/jira/browse/HBASE-18043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18043: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Institute a hard limit for individual cell size that cannot be overridden by > clients > > > Key: HBASE-18043 > URL: https://issues.apache.org/jira/browse/HBASE-18043 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, regionserver >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-18043-branch-1.patch, HBASE-18043-branch-1.patch, > HBASE-18043-branch-1.patch, HBASE-18043-branch-1.patch, HBASE-18043.patch, > HBASE-18043.patch, HBASE-18043.patch, HBASE-18043.patch, HBASE-18043.patch, > HBASE-18043.patch > > > For sake of service protection we should not give absolute trust to clients > regarding resource limits that can impact stability, like cell size limits. > We should add a server side configuration that sets a hard limit for > individual cell size that cannot be overridden by the client. We can keep the > client side check, because it's expensive to reject a RPC that has already > come in. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18026) ProtobufUtil seems to do extra array copying
[ https://issues.apache.org/jira/browse/HBASE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18026: --- Affects Version/s: (was: 2.0.0) Fix Version/s: (was: 2.0.0) > ProtobufUtil seems to do extra array copying > > > Key: HBASE-18026 > URL: https://issues.apache.org/jira/browse/HBASE-18026 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-18026.addendum.patch, > HBASE-18026.branch-1.v1.patch, HBASE-18026.master.v1.patch > > > In ProtobufUtil, the protobuf fields are copied into an array using > toByteArray(). These are then passed into the KeyValue constructor which > does another copy. > It seems like we can avoid a copy here by using > HBaseZeroCopyByteString#zeroCopyGetBytes() ? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17359) Implement async admin
[ https://issues.apache.org/jira/browse/HBASE-17359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011595#comment-16011595 ] Duo Zhang commented on HBASE-17359: --- So what's the status of this issue? Have we done? [~zghaobac] [~openinx]. > Implement async admin > - > > Key: HBASE-17359 > URL: https://issues.apache.org/jira/browse/HBASE-17359 > Project: HBase > Issue Type: Umbrella > Components: Client >Reporter: Duo Zhang >Assignee: Guanghao Zhang > Labels: asynchronous > Fix For: 2.0.0 > > > And as we will return a CompletableFuture, I think we can just remove the > XXXAsync methods, and make all the methods blocking which means we will only > finish the CompletableFuture when the operation is done. User can choose > whether to wait on the returned CompletableFuture. > Convert this to a umbrella task. There maybe some sub-tasks. > 1. Table admin operations. > 2. Region admin operations. > 3. Namespace admin operations. > 4. Snapshot admin operations. > 5. Replication admin operations. > 6. Other operations, like quota, balance.. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18038) Rename StoreFile to HStoreFile and add a StoreFile interface for CP
[ https://issues.apache.org/jira/browse/HBASE-18038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-18038: -- Attachment: HBASE-18038-v1.patch Retry. > Rename StoreFile to HStoreFile and add a StoreFile interface for CP > --- > > Key: HBASE-18038 > URL: https://issues.apache.org/jira/browse/HBASE-18038 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, regionserver >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-18038.patch, HBASE-18038-v1.patch, > HBASE-18038-v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18035) Meta replica does not give any primaryOperationTimeout to primary meta region
[ https://issues.apache.org/jira/browse/HBASE-18035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011587#comment-16011587 ] huaxiang sun commented on HBASE-18035: -- As a side note, why is memstore replication not turned on for meta replica? Probably will create a jira to track it. > Meta replica does not give any primaryOperationTimeout to primary meta region > - > > Key: HBASE-18035 > URL: https://issues.apache.org/jira/browse/HBASE-18035 > Project: HBase > Issue Type: Bug >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Critical > Attachments: 18035-unittest.patch, HBASE-18035-master-v001.patch > > > I was working on my unittest and it failed with TableNotFoundException. I > debugged a bit and found out that for meta scan, it does not give any > primaryOperationTimeout to primary meta region. This will be an issue as the > meta replica will contain stale data and it is possible that the meta replica > will return back first than primary. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java#L823 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18035) Meta replica does not give any primaryOperationTimeout to primary meta region
[ https://issues.apache.org/jira/browse/HBASE-18035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-18035: - Attachment: HBASE-18035-master-v001.patch Upload v1 patch. > Meta replica does not give any primaryOperationTimeout to primary meta region > - > > Key: HBASE-18035 > URL: https://issues.apache.org/jira/browse/HBASE-18035 > Project: HBase > Issue Type: Bug >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Critical > Attachments: 18035-unittest.patch, HBASE-18035-master-v001.patch > > > I was working on my unittest and it failed with TableNotFoundException. I > debugged a bit and found out that for meta scan, it does not give any > primaryOperationTimeout to primary meta region. This will be an issue as the > meta replica will contain stale data and it is possible that the meta replica > will return back first than primary. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java#L823 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18035) Meta replica does not give any primaryOperationTimeout to primary meta region
[ https://issues.apache.org/jira/browse/HBASE-18035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-18035: - Status: Patch Available (was: Open) > Meta replica does not give any primaryOperationTimeout to primary meta region > - > > Key: HBASE-18035 > URL: https://issues.apache.org/jira/browse/HBASE-18035 > Project: HBase > Issue Type: Bug >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Critical > Attachments: 18035-unittest.patch, HBASE-18035-master-v001.patch > > > I was working on my unittest and it failed with TableNotFoundException. I > debugged a bit and found out that for meta scan, it does not give any > primaryOperationTimeout to primary meta region. This will be an issue as the > meta replica will contain stale data and it is possible that the meta replica > will return back first than primary. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java#L823 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18037) Do not expose implementation classes to CP
[ https://issues.apache.org/jira/browse/HBASE-18037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011574#comment-16011574 ] Andrew Purtell commented on HBASE-18037: bq. you are free to hold a reference to a class which is declared as IA.Private, but you can only use it as a monolithic object. You can use it as a parameter to call a method, or use it as a return value, but you should never try to call its methods to change its state of do something else. Sounds good [~Apache9]. I see you filed HBASE-18050 for followup > Do not expose implementation classes to CP > -- > > Key: HBASE-18037 > URL: https://issues.apache.org/jira/browse/HBASE-18037 > Project: HBase > Issue Type: Umbrella > Components: Coprocessors >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Priority: Blocker > Fix For: 2.0.0 > > > For example, StoreFile. Expose the implementation classes to CP will make it > harder to implement new features or improve the old implementation. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-18026) ProtobufUtil seems to do extra array copying
[ https://issues.apache.org/jira/browse/HBASE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011570#comment-16011570 ] Andrew Purtell edited comment on HBASE-18026 at 5/16/17 1:10 AM: - bq. I suggest revering the change in master branch as any way it is not useful and needed. For other branches, I also suggest adding the addendum Ok [~anoop.hbase] - doing that now was (Author: apurtell): Ok [~anoop.hbase] - doing that now > ProtobufUtil seems to do extra array copying > > > Key: HBASE-18026 > URL: https://issues.apache.org/jira/browse/HBASE-18026 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 2.0.0, 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-18026.addendum.patch, > HBASE-18026.branch-1.v1.patch, HBASE-18026.master.v1.patch > > > In ProtobufUtil, the protobuf fields are copied into an array using > toByteArray(). These are then passed into the KeyValue constructor which > does another copy. > It seems like we can avoid a copy here by using > HBaseZeroCopyByteString#zeroCopyGetBytes() ? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18026) ProtobufUtil seems to do extra array copying
[ https://issues.apache.org/jira/browse/HBASE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011570#comment-16011570 ] Andrew Purtell commented on HBASE-18026: Ok [~anoop.hbase] - doing that now > ProtobufUtil seems to do extra array copying > > > Key: HBASE-18026 > URL: https://issues.apache.org/jira/browse/HBASE-18026 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.2 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Minor > Fix For: 2.0.0, 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-18026.addendum.patch, > HBASE-18026.branch-1.v1.patch, HBASE-18026.master.v1.patch > > > In ProtobufUtil, the protobuf fields are copied into an array using > toByteArray(). These are then passed into the KeyValue constructor which > does another copy. > It seems like we can avoid a copy here by using > HBaseZeroCopyByteString#zeroCopyGetBytes() ? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18027) Replication should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18027: --- Status: Patch Available (was: Open) > Replication should respect RPC size limits when batching edits > -- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.3.1, 2.0.0, 1.4.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18044) Bug fix in flaky-dashboard-template
[ https://issues.apache.org/jira/browse/HBASE-18044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011521#comment-16011521 ] Hudson commented on HBASE-18044: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #3017 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3017/]) HBASE-18044 Fix bug in report-flakies.py where if counter is set outside (appy: rev 841bb00655b3eb58e7d76d945e24cfeadfdb11de) * (edit) dev-support/flaky-dashboard-template.html > Bug fix in flaky-dashboard-template > --- > > Key: HBASE-18044 > URL: https://issues.apache.org/jira/browse/HBASE-18044 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-18044.master.001.patch > > > Due to some scoping issues, counters don't work in jinja templates like we > expect them to (in java/c++,etc). > (http://stackoverflow.com/questions/7537439/how-to-increment-a-variable-on-a-for-loop-in-jinja-template). > Due to this, clicking show/hide button for tests which fail in multiple urls > will not work sometime. Fixing it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18035) Meta replica does not give any primaryOperationTimeout to primary meta region
[ https://issues.apache.org/jira/browse/HBASE-18035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011517#comment-16011517 ] huaxiang sun commented on HBASE-18035: -- I am planning to propose add "hbase.client.metaReplicaCallTimeout.scan" for primary meta region scan. > Meta replica does not give any primaryOperationTimeout to primary meta region > - > > Key: HBASE-18035 > URL: https://issues.apache.org/jira/browse/HBASE-18035 > Project: HBase > Issue Type: Bug >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Critical > Attachments: 18035-unittest.patch > > > I was working on my unittest and it failed with TableNotFoundException. I > debugged a bit and found out that for meta scan, it does not give any > primaryOperationTimeout to primary meta region. This will be an issue as the > meta replica will contain stale data and it is possible that the meta replica > will return back first than primary. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java#L823 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18043) Institute a hard limit for individual cell size that cannot be overridden by clients
[
https://issues.apache.org/jira/browse/HBASE-18043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011510#comment-16011510
]
Hadoop QA commented on HBASE-18043:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 23s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 30s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
32m 21s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 45s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 52s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
30s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 171m 43s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestMobStoreScanner |
| | hadoop.hbase.mapreduce.TestImportTSVWithVisibilityLabels |
| | hadoop.hbase.master.TestMasterMetrics |
| | hadoop.hbase.snapshot.TestMobFlushSnapshotFromClient |
| | hadoop.hbase.quotas.TestQuotaThrottle |
| Timed out junit tests |
org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure |
| | org.apache.hadoop.hbase.master.TestMasterFailover |
| | org.apache.hadoop.hbase.regionserver.TestTags |
| | org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster |
| | org.apache.hadoop.hbase.master.balancer.TestFavoredStochasticLoadBalancer
|
| | org.apache.hadoop.hbase.master.balancer.TestFavoredNodeTableImport |
| | org.apache.hadoop.hbase.mapreduce.TestTableMapReduce |
| | org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestExportSnapshot |
|
[jira] [Updated] (HBASE-18035) Meta replica does not give any primaryOperationTimeout to primary meta region
[ https://issues.apache.org/jira/browse/HBASE-18035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-18035: - Attachment: (was: 18035-unittest.diff) > Meta replica does not give any primaryOperationTimeout to primary meta region > - > > Key: HBASE-18035 > URL: https://issues.apache.org/jira/browse/HBASE-18035 > Project: HBase > Issue Type: Bug >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Critical > Attachments: 18035-unittest.patch > > > I was working on my unittest and it failed with TableNotFoundException. I > debugged a bit and found out that for meta scan, it does not give any > primaryOperationTimeout to primary meta region. This will be an issue as the > meta replica will contain stale data and it is possible that the meta replica > will return back first than primary. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java#L823 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18035) Meta replica does not give any primaryOperationTimeout to primary meta region
[ https://issues.apache.org/jira/browse/HBASE-18035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-18035: - Attachment: 18035-unittest.patch > Meta replica does not give any primaryOperationTimeout to primary meta region > - > > Key: HBASE-18035 > URL: https://issues.apache.org/jira/browse/HBASE-18035 > Project: HBase > Issue Type: Bug >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Critical > Attachments: 18035-unittest.patch > > > I was working on my unittest and it failed with TableNotFoundException. I > debugged a bit and found out that for meta scan, it does not give any > primaryOperationTimeout to primary meta region. This will be an issue as the > meta replica will contain stale data and it is possible that the meta replica > will return back first than primary. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java#L823 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18035) Meta replica does not give any primaryOperationTimeout to primary meta region
[ https://issues.apache.org/jira/browse/HBASE-18035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-18035: - Attachment: 18035-unittest.diff Upload a unittest case to demo the issue. > Meta replica does not give any primaryOperationTimeout to primary meta region > - > > Key: HBASE-18035 > URL: https://issues.apache.org/jira/browse/HBASE-18035 > Project: HBase > Issue Type: Bug >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Critical > Attachments: 18035-unittest.diff > > > I was working on my unittest and it failed with TableNotFoundException. I > debugged a bit and found out that for meta scan, it does not give any > primaryOperationTimeout to primary meta region. This will be an issue as the > meta replica will contain stale data and it is possible that the meta replica > will return back first than primary. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java#L823 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-18027) Replication should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011484#comment-16011484 ] Andrew Purtell edited comment on HBASE-18027 at 5/15/17 11:13 PM: -- Ok, here is a patch that moves the discussed concerns up into the caller(s). This is in response to feedback from [~lhofhansl]. Although inside HBaseInterClusterReplicationEndpoint the batch of edits may be broken up into smaller batches for parallelization, up in the callers we can't see how the sub-batches might be partitioned. So, we have to consider the RPC request limit with respect to the whole batch, which does simplify the change: If the replication batch capacity will exceed the RPC request limit we should simply use the RPC request limit as the replication batch capacity. The downside is this is more pessimistic than if we check limits right where we are building the sub-batches and know the size of each. On the other hand it is easier to understand how the respective configuration parameters will interact. Changes: - Where we set replicationBatchSizeCapacity in ReplicationSourceWALReaderThread, check if replicationBatchSizeCapacity will exceed the RPC request size limit, and if so use the request size limit instead and warn about it with mention of relevant configuration keys. - When building the current replication batch, check if we will exceed the batch size capacity _before_ adding an entry to the batch. Ensure at least one entry is added to the batch even if it means we violate quota or batch size capacity or otherwise replication would become stuck. - WALEntryStream needs a putBack() method to undo next() when we've decided the entry we just iterated to will cause the batch to exceed size limits. and minor changes to logging: - New debug logging in HBaseInterClusterReplicationEndpoint. One new DEBUG level line per replication batch. - Fix trace level log that conflated total amount of data in replication context with size of data in the worklist submitted as Replicator runnable. - Add details to the warning logged if the number of edits replicated is different from the number received. All replication unit tests pass with these changes applied. was (Author: apurtell): Ok, here is a patch that moves the discussed concerns up into the caller(s). This is in response to feedback from [~lhofhansl]. Although inside HBaseInterClusterReplicationEndpoint the batch of edits may be broken up into smaller batches for parallelization, up in the callers we can't see how the sub-batches might be partitioned. So, we have to consider the RPC request limit with respect to the whole batch, which does simplify the change: If the replication batch capacity will exceed the RPC request limit we should simply use the RPC request limit as the replication batch capacity. The downside is this is more pessimistic than if we check limits right where we are building the sub-batches and know the size of each. Changes: - Where we set replicationBatchSizeCapacity in ReplicationSourceWALReaderThread, check if replicationBatchSizeCapacity will exceed the RPC request size limit, and if so use the request size limit instead and warn about it with mention of relevant configuration keys. - When building the current replication batch, check if we will exceed the batch size capacity _before_ adding an entry to the batch. Ensure at least one entry is added to the batch even if it means we violate quota or batch size capacity or otherwise replication would become stuck. - WALEntryStream needs a putBack() method to undo next() when we've decided the entry we just iterated to will cause the batch to exceed size limits. and minor changes to logging: - New debug logging in HBaseInterClusterReplicationEndpoint. One new DEBUG level line per replication batch. - Fix trace level log that conflated total amount of data in replication context with size of data in the worklist submitted as Replicator runnable. - Add details to the warning logged if the number of edits replicated is different from the number received. All replication unit tests pass with these changes applied. > Replication should respect RPC size limits when batching edits > -- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum
[jira] [Updated] (HBASE-18027) Replication should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18027: --- Attachment: HBASE-18027.patch Ok, here is a patch that moves the discussed concerns up into the caller(s). This is in response to feedback from [~lhofhansl]. Although inside HBaseInterClusterReplicationEndpoint the batch of edits may be broken up into smaller batches for parallelization, up in the callers we can't see how the sub-batches might be partitioned. So, we have to consider the RPC request limit with respect to the whole batch, which does simplify the change: If the replication batch capacity will exceed the RPC request limit we should simply use the RPC request limit as the replication batch capacity. The downside is this is more pessimistic than if we check limits right where we are building the sub-batches and know the size of each. Changes: - Where we set replicationBatchSizeCapacity in ReplicationSourceWALReaderThread, check if replicationBatchSizeCapacity will exceed the RPC request size limit, and if so use the request size limit instead and warn about it with mention of relevant configuration keys. - When building the current replication batch, check if we will exceed the batch size capacity _before_ adding an entry to the batch. Ensure at least one entry is added to the batch even if it means we violate quota or batch size capacity or otherwise replication would become stuck. - WALEntryStream needs a putBack() method to undo next() when we've decided the entry we just iterated to will cause the batch to exceed size limits. and minor changes to logging: - New debug logging in HBaseInterClusterReplicationEndpoint. One new DEBUG level line per replication batch. - Fix trace level log that conflated total amount of data in replication context with size of data in the worklist submitted as Replicator runnable. - Add details to the warning logged if the number of edits replicated is different from the number received. All *Replication* unit tests pass with these changes applied. > Replication should respect RPC size limits when batching edits > -- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-18027) Replication should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011484#comment-16011484 ] Andrew Purtell edited comment on HBASE-18027 at 5/15/17 11:12 PM: -- Ok, here is a patch that moves the discussed concerns up into the caller(s). This is in response to feedback from [~lhofhansl]. Although inside HBaseInterClusterReplicationEndpoint the batch of edits may be broken up into smaller batches for parallelization, up in the callers we can't see how the sub-batches might be partitioned. So, we have to consider the RPC request limit with respect to the whole batch, which does simplify the change: If the replication batch capacity will exceed the RPC request limit we should simply use the RPC request limit as the replication batch capacity. The downside is this is more pessimistic than if we check limits right where we are building the sub-batches and know the size of each. Changes: - Where we set replicationBatchSizeCapacity in ReplicationSourceWALReaderThread, check if replicationBatchSizeCapacity will exceed the RPC request size limit, and if so use the request size limit instead and warn about it with mention of relevant configuration keys. - When building the current replication batch, check if we will exceed the batch size capacity _before_ adding an entry to the batch. Ensure at least one entry is added to the batch even if it means we violate quota or batch size capacity or otherwise replication would become stuck. - WALEntryStream needs a putBack() method to undo next() when we've decided the entry we just iterated to will cause the batch to exceed size limits. and minor changes to logging: - New debug logging in HBaseInterClusterReplicationEndpoint. One new DEBUG level line per replication batch. - Fix trace level log that conflated total amount of data in replication context with size of data in the worklist submitted as Replicator runnable. - Add details to the warning logged if the number of edits replicated is different from the number received. All replication unit tests pass with these changes applied. was (Author: apurtell): Ok, here is a patch that moves the discussed concerns up into the caller(s). This is in response to feedback from [~lhofhansl]. Although inside HBaseInterClusterReplicationEndpoint the batch of edits may be broken up into smaller batches for parallelization, up in the callers we can't see how the sub-batches might be partitioned. So, we have to consider the RPC request limit with respect to the whole batch, which does simplify the change: If the replication batch capacity will exceed the RPC request limit we should simply use the RPC request limit as the replication batch capacity. The downside is this is more pessimistic than if we check limits right where we are building the sub-batches and know the size of each. Changes: - Where we set replicationBatchSizeCapacity in ReplicationSourceWALReaderThread, check if replicationBatchSizeCapacity will exceed the RPC request size limit, and if so use the request size limit instead and warn about it with mention of relevant configuration keys. - When building the current replication batch, check if we will exceed the batch size capacity _before_ adding an entry to the batch. Ensure at least one entry is added to the batch even if it means we violate quota or batch size capacity or otherwise replication would become stuck. - WALEntryStream needs a putBack() method to undo next() when we've decided the entry we just iterated to will cause the batch to exceed size limits. and minor changes to logging: - New debug logging in HBaseInterClusterReplicationEndpoint. One new DEBUG level line per replication batch. - Fix trace level log that conflated total amount of data in replication context with size of data in the worklist submitted as Replicator runnable. - Add details to the warning logged if the number of edits replicated is different from the number received. All *Replication* unit tests pass with these changes applied. > Replication should respect RPC size limits when batching edits > -- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every
[jira] [Commented] (HBASE-18042) Client Compatibility breaks between versions 1.2 and 1.3
[
https://issues.apache.org/jira/browse/HBASE-18042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011477#comment-16011477
]
Duo Zhang commented on HBASE-18042:
---
The field has been introduced in all branches, even for 0.98. So I think the
fix for AsyncHBase is straightforward? If the field is present then depend on
the value of this field to decide if we need to go to the next region. If not,
use the old logic.
And we also need a fix for all branches below 1.3 I think. If this field is
present then we should not issue a new next request then. We can do it here or
open a new issue.
Thanks.
> Client Compatibility breaks between versions 1.2 and 1.3
>
>
> Key: HBASE-18042
> URL: https://issues.apache.org/jira/browse/HBASE-18042
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.1
>Reporter: Karan Mehta
>Assignee: Karan Mehta
>
> OpenTSDB uses AsyncHBase as its client, rather than using the traditional
> HBase Client. From version 1.2 to 1.3, the {{ClientProtos}} have been
> changed. Newer fields are added to {{ScanResponse}} proto.
> For a typical Scan request in 1.2, would require caller to make an
> OpenScanner Request, GetNextRows Request and a CloseScanner Request, based on
> {{more_rows}} boolean field in the {{ScanResponse}} proto.
> However, from 1.3, new parameter {{more_results_in_region}} was added, which
> limits the results per region. Therefore the client has to now manage sending
> all the requests for each region. Further more, if the results are exhausted
> from a particular region, the {{ScanResponse}} will set
> {{more_results_in_region}} to false, but {{more_results}} can still be true.
> Whenever the former is set to false, the {{RegionScanner}} will also be
> closed.
> OpenTSDB makes an OpenScanner Request and receives all its results in the
> first {{ScanResponse}} itself, thus creating a condition as described in
> above paragraph. Since {{more_rows}} is true, it will proceed to send next
> request at which point the {{RSRpcServices}} will throw
> {{UnknownScannerException}}. The protobuf client compatibility is maintained
> but expected behavior is modified.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18005) read replica: handle the case that region server hosting both primary replica and meta region is down
[
https://issues.apache.org/jira/browse/HBASE-18005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011469#comment-16011469
]
Hadoop QA commented on HBASE-18005:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 2s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
34m 20s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
33s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 20s
{color} | {color:red} hbase-client generated 1 new + 1 unchanged - 0 fixed = 2
total (was 1) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 41s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 135m 39s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 215m 19s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12868131/HBASE-18005-master-002.patch
|
| JIRA Issue | HBASE-18005 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux ab45fba779fb 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 341223d |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/6793/artifact/patchprocess/whitespace-eol.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-HBASE-Build/6793/artifact/patchprocess/diff-javadoc-javadoc-hbase-client.txt
|
| Test Results |
https://build
[jira] [Commented] (HBASE-15370) Backport Moderate Object Storage (MOB) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-15370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011412#comment-16011412 ] Sean Busbey commented on HBASE-15370: - FWIW, I wouldn't say that thread has run its course yet. I agree that it looks like things are bending towards no backport. > Backport Moderate Object Storage (MOB) to branch-1 > -- > > Key: HBASE-15370 > URL: https://issues.apache.org/jira/browse/HBASE-15370 > Project: HBase > Issue Type: Task >Reporter: Ted Yu > Fix For: 1.4.0 > > Attachments: 15370-test.out, HBASE-15370-branch-1.v1.patch, > HBASE-15370-branch-1.v2.patch, merge-conflict.list, mob-cmmits.txt, > mob-commits-v2.txt > > > MOB feature was integrated to master branch half a year ago. > Since then there have been bug fixes which stabilize the feature. > Some customers have been using it at PB scale. > Here is discussion thread on dev mailing list: > http://search-hadoop.com/m/YGbbDSqxD1PYXK62/hbase+MOB+in+branch-1&subj=Re+MOB+in+branch+1+Re+RESULT+VOTE+Merge+branch+hbase+11339+HBase+MOB+to+trunk+ > This issue is to backport MOB feature to branch-1. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15370) Backport Moderate Object Storage (MOB) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-15370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011402#comment-16011402 ] Ted Yu commented on HBASE-15370: This is recent thread on backporting where consensus is not to backport: http://search-hadoop.com/m/HBase/YGbb2NdxC1okisI1?subj=Re+HBASE+and+MOB > Backport Moderate Object Storage (MOB) to branch-1 > -- > > Key: HBASE-15370 > URL: https://issues.apache.org/jira/browse/HBASE-15370 > Project: HBase > Issue Type: Task >Reporter: Ted Yu > Fix For: 1.4.0 > > Attachments: 15370-test.out, HBASE-15370-branch-1.v1.patch, > HBASE-15370-branch-1.v2.patch, merge-conflict.list, mob-cmmits.txt, > mob-commits-v2.txt > > > MOB feature was integrated to master branch half a year ago. > Since then there have been bug fixes which stabilize the feature. > Some customers have been using it at PB scale. > Here is discussion thread on dev mailing list: > http://search-hadoop.com/m/YGbbDSqxD1PYXK62/hbase+MOB+in+branch-1&subj=Re+MOB+in+branch+1+Re+RESULT+VOTE+Merge+branch+hbase+11339+HBase+MOB+to+trunk+ > This issue is to backport MOB feature to branch-1. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-18042) Client Compatibility breaks between versions 1.2 and 1.3
[
https://issues.apache.org/jira/browse/HBASE-18042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011284#comment-16011284
]
Andrew Purtell edited comment on HBASE-18042 at 5/15/17 9:49 PM:
-
We are not supposed to break wire compatibility in minor versions. This is
arguable. The protobuf change is compatible in the strictest sense in that
mismatched versions can still parse messages. The fact that semantics for
handling a given field has changed between older and newer versions, such that
mishandling causes exceptions and scanner failure, means not really.
was (Author: apurtell):
We are not supposed to break wire compatibility in minor versions. This is
arguable. The protobuf change is compatible in the strictest sense in that
mismatched versions can still parse messages. The fact that semantics rely on a
new field not present in older versions means not really.
> Client Compatibility breaks between versions 1.2 and 1.3
>
>
> Key: HBASE-18042
> URL: https://issues.apache.org/jira/browse/HBASE-18042
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.1
>Reporter: Karan Mehta
>Assignee: Karan Mehta
>
> OpenTSDB uses AsyncHBase as its client, rather than using the traditional
> HBase Client. From version 1.2 to 1.3, the {{ClientProtos}} have been
> changed. Newer fields are added to {{ScanResponse}} proto.
> For a typical Scan request in 1.2, would require caller to make an
> OpenScanner Request, GetNextRows Request and a CloseScanner Request, based on
> {{more_rows}} boolean field in the {{ScanResponse}} proto.
> However, from 1.3, new parameter {{more_results_in_region}} was added, which
> limits the results per region. Therefore the client has to now manage sending
> all the requests for each region. Further more, if the results are exhausted
> from a particular region, the {{ScanResponse}} will set
> {{more_results_in_region}} to false, but {{more_results}} can still be true.
> Whenever the former is set to false, the {{RegionScanner}} will also be
> closed.
> OpenTSDB makes an OpenScanner Request and receives all its results in the
> first {{ScanResponse}} itself, thus creating a condition as described in
> above paragraph. Since {{more_rows}} is true, it will proceed to send next
> request at which point the {{RSRpcServices}} will throw
> {{UnknownScannerException}}. The protobuf client compatibility is maintained
> but expected behavior is modified.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18027) Replication should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18027: --- Summary: Replication should respect RPC size limits when batching edits (was: HBaseInterClusterReplicationEndpoint should respect RPC size limits when batching edits) > Replication should respect RPC size limits when batching edits > -- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18044) Bug fix in flaky-dashboard-template
[ https://issues.apache.org/jira/browse/HBASE-18044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011344#comment-16011344 ] Appy commented on HBASE-18044: -- Thanks [~stack]. Your suggestion would surely make it easier for me to fix small things in our dev-support. > Bug fix in flaky-dashboard-template > --- > > Key: HBASE-18044 > URL: https://issues.apache.org/jira/browse/HBASE-18044 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-18044.master.001.patch > > > Due to some scoping issues, counters don't work in jinja templates like we > expect them to (in java/c++,etc). > (http://stackoverflow.com/questions/7537439/how-to-increment-a-variable-on-a-for-loop-in-jinja-template). > Due to this, clicking show/hide button for tests which fail in multiple urls > will not work sometime. Fixing it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18044) Bug fix in flaky-dashboard-template
[ https://issues.apache.org/jira/browse/HBASE-18044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-18044: - Resolution: Fixed Status: Resolved (was: Patch Available) > Bug fix in flaky-dashboard-template > --- > > Key: HBASE-18044 > URL: https://issues.apache.org/jira/browse/HBASE-18044 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-18044.master.001.patch > > > Due to some scoping issues, counters don't work in jinja templates like we > expect them to (in java/c++,etc). > (http://stackoverflow.com/questions/7537439/how-to-increment-a-variable-on-a-for-loop-in-jinja-template). > Due to this, clicking show/hide button for tests which fail in multiple urls > will not work sometime. Fixing it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18042) Client Compatibility breaks between versions 1.2 and 1.3
[
https://issues.apache.org/jira/browse/HBASE-18042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011327#comment-16011327
]
Benoit Sigoure commented on HBASE-18042:
We can easily update AsyncHBase to accommodate to the change however I would
like to voice disagreement with this statement:
{quote}
It is an unfortunate thing that we have broken the semantics, but in general
this is "allowed".
{quote}
Such semantic changes are like breaking API changes, they are, well, breaking
changes. Not cool.
One of the challenges with AsyncHBase is that it has to work with all versions
of HBase. Since {{more_results_in_region}} was already there in 1.2 but needs
to be handled differently in 1.3, that makes it kinda hard for AsyncHBase to
know how, exactly, to deal with this flag being set, right?
> Client Compatibility breaks between versions 1.2 and 1.3
>
>
> Key: HBASE-18042
> URL: https://issues.apache.org/jira/browse/HBASE-18042
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.1
>Reporter: Karan Mehta
>Assignee: Karan Mehta
>
> OpenTSDB uses AsyncHBase as its client, rather than using the traditional
> HBase Client. From version 1.2 to 1.3, the {{ClientProtos}} have been
> changed. Newer fields are added to {{ScanResponse}} proto.
> For a typical Scan request in 1.2, would require caller to make an
> OpenScanner Request, GetNextRows Request and a CloseScanner Request, based on
> {{more_rows}} boolean field in the {{ScanResponse}} proto.
> However, from 1.3, new parameter {{more_results_in_region}} was added, which
> limits the results per region. Therefore the client has to now manage sending
> all the requests for each region. Further more, if the results are exhausted
> from a particular region, the {{ScanResponse}} will set
> {{more_results_in_region}} to false, but {{more_results}} can still be true.
> Whenever the former is set to false, the {{RegionScanner}} will also be
> closed.
> OpenTSDB makes an OpenScanner Request and receives all its results in the
> first {{ScanResponse}} itself, thus creating a condition as described in
> above paragraph. Since {{more_rows}} is true, it will proceed to send next
> request at which point the {{RSRpcServices}} will throw
> {{UnknownScannerException}}. The protobuf client compatibility is maintained
> but expected behavior is modified.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18043) Institute a hard limit for individual cell size that cannot be overridden by clients
[ https://issues.apache.org/jira/browse/HBASE-18043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18043: --- Attachment: HBASE-18043.patch > Institute a hard limit for individual cell size that cannot be overridden by > clients > > > Key: HBASE-18043 > URL: https://issues.apache.org/jira/browse/HBASE-18043 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, regionserver >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-18043-branch-1.patch, HBASE-18043-branch-1.patch, > HBASE-18043-branch-1.patch, HBASE-18043-branch-1.patch, HBASE-18043.patch, > HBASE-18043.patch, HBASE-18043.patch, HBASE-18043.patch, HBASE-18043.patch, > HBASE-18043.patch > > > For sake of service protection we should not give absolute trust to clients > regarding resource limits that can impact stability, like cell size limits. > We should add a server side configuration that sets a hard limit for > individual cell size that cannot be overridden by the client. We can keep the > client side check, because it's expensive to reject a RPC that has already > come in. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
