[jira] [Updated] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
[ https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15525: --- Attachment: HBASE-15525_V2.patch > OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts > -- > > Key: HBASE-15525 > URL: https://issues.apache.org/jira/browse/HBASE-15525 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: deepankar >Assignee: Anoop Sam John >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-15525_V1.patch, HBASE-15525_V2.patch, > HBASE-15525_WIP.patch, WIP.patch > > > After HBASE-13819 the system some times run out of direct memory whenever > there is some network congestion or some client side issues. > This was because of pending RPCs in the RPCServer$Connection.responseQueue > and since all the responses in this queue hold a buffer for cellblock from > BoundedByteBufferPool this could takeup a lot of memory if the > BoundedByteBufferPool's moving average settles down towards a higher value > See the discussion here > [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15594) [YCSB] Improvements
[
https://issues.apache.org/jira/browse/HBASE-15594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227763#comment-15227763
]
Duo Zhang commented on HBASE-15594:
---
I do not have the permission to change the BIOS option of a server either...
But I think hyper-threading is a general option that does not only effect
HBase. We could try some other applications on our own machine with or without
hyper-threading.
What do you think? [~stack]
> [YCSB] Improvements
> ---
>
> Key: HBASE-15594
> URL: https://issues.apache.org/jira/browse/HBASE-15594
> Project: HBase
> Issue Type: Umbrella
>Reporter: stack
>Priority: Critical
>
> Running YCSB and getting good results is an arcane art. For example, in my
> testing, a few handlers (100) with as many readers as I had CPUs (48), and
> upping connections on clients to same as #cpus made for 2-3x the throughput.
> The above config changes came of lore; which configurations need tweaking is
> not obvious going by their names, there were no indications from the app on
> where/why we were blocked or on which metrics are important to consider. Nor
> was any of this stuff written down in docs.
> Even still, I am stuck trying to make use of all of the machine. I am unable
> to overrun a server though 8 client nodes trying to beat up a single node
> (workloadc, all random-read, with no data returned -p readallfields=false).
> There is also a strange phenomenon where if I add a few machines, rather than
> 3x the YCSB throughput when 3 nodes in cluster, each machine instead is doing
> about 1/3rd.
> This umbrella issue is to host items that improve our defaults and noting how
> to get good numbers running YCSB. In particular, I want to be able to
> saturate a machine.
> Here are the configs I'm currently working with. I've not done the work to
> figure client-side if they are optimal (weird is how big a difference
> client-side changes can make -- need to fix this). On my 48 cpu machine, I
> can do about 370k random reads a second from data totally cached in
> bucketcache. If I short-circuit the user gets so they don't do any work but
> return immediately, I can do 600k ops a second but the CPUs are at 60-70%
> only. I cannot get them to go above this. Working on it.
> {code}
>
>
> hbase.ipc.server.read.threadpool.size
>
> 48
>
>
>
> hbase.regionserver.handler.count
>
> 100
>
>
>
> hbase.client.ipc.pool.size
>
> 100
>
>
>
> hbase.htable.threads.max
>
> 48
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15594) [YCSB] Improvements
[
https://issues.apache.org/jira/browse/HBASE-15594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227753#comment-15227753
]
stack commented on HBASE-15594:
---
Thanks [~Apache9] for chiming in. I don't have easy access to the machines so a
bit of a PITA disabling hyperthreading. Does anyone actually do this on
production machines? Do you fellows do it? Thanks.
> [YCSB] Improvements
> ---
>
> Key: HBASE-15594
> URL: https://issues.apache.org/jira/browse/HBASE-15594
> Project: HBase
> Issue Type: Umbrella
>Reporter: stack
>Priority: Critical
>
> Running YCSB and getting good results is an arcane art. For example, in my
> testing, a few handlers (100) with as many readers as I had CPUs (48), and
> upping connections on clients to same as #cpus made for 2-3x the throughput.
> The above config changes came of lore; which configurations need tweaking is
> not obvious going by their names, there were no indications from the app on
> where/why we were blocked or on which metrics are important to consider. Nor
> was any of this stuff written down in docs.
> Even still, I am stuck trying to make use of all of the machine. I am unable
> to overrun a server though 8 client nodes trying to beat up a single node
> (workloadc, all random-read, with no data returned -p readallfields=false).
> There is also a strange phenomenon where if I add a few machines, rather than
> 3x the YCSB throughput when 3 nodes in cluster, each machine instead is doing
> about 1/3rd.
> This umbrella issue is to host items that improve our defaults and noting how
> to get good numbers running YCSB. In particular, I want to be able to
> saturate a machine.
> Here are the configs I'm currently working with. I've not done the work to
> figure client-side if they are optimal (weird is how big a difference
> client-side changes can make -- need to fix this). On my 48 cpu machine, I
> can do about 370k random reads a second from data totally cached in
> bucketcache. If I short-circuit the user gets so they don't do any work but
> return immediately, I can do 600k ops a second but the CPUs are at 60-70%
> only. I cannot get them to go above this. Working on it.
> {code}
>
>
> hbase.ipc.server.read.threadpool.size
>
> 48
>
>
>
> hbase.regionserver.handler.count
>
> 100
>
>
>
> hbase.client.ipc.pool.size
>
> 100
>
>
>
> hbase.htable.threads.max
>
> 48
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15592) Print Procedure WAL content
[
https://issues.apache.org/jira/browse/HBASE-15592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227732#comment-15227732
]
Hadoop QA commented on HBASE-15592:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 16s
{color} | {color:red} hbase-procedure in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 28s
{color} | {color:red} hbase-procedure in the patch failed with JDK v1.8.0.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 28s {color}
| {color:red} hbase-procedure in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 15s
{color} | {color:red} hbase-procedure in the patch failed with JDK v1.7.0_79.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 15s {color}
| {color:red} hbase-procedure in the patch failed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 26s
{color} | {color:red} hbase-procedure: patch generated 1 new + 2 unchanged - 0
fixed = 3 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 54s
{color} | {color:red} Patch causes 14 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 54s
{color} | {color:red} Patch causes 14 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 49s
{color} | {color:red} Patch causes 14 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 36s
{color} | {color:red} Patch causes 14 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 27s
{color} | {color:red} Patch causes 14 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 21s
{color} | {color:red} Patch causes 14 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 16s
{color} | {color:red} Patch causes 14 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m 11s
{color} | {color:red} Patch causes 14 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 1s
{color} | {color:red} Patch causes 14 errors with Hadoop v2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 15s
{color} | {color:red} hbase-procedure in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
[jira] [Commented] (HBASE-15591) ServerCrashProcedure not yielding
[ https://issues.apache.org/jira/browse/HBASE-15591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227730#comment-15227730 ] Matteo Bertozzi commented on HBASE-15591: - a yield is supposed to let someone else run. so adding a sleep on yield is not a valid generic solution. The idea is that in similar cases where you have to wait on something you go in a waiting queue until you get the event. but we still not have everything to implement that, so at the moment this procedure does not follow the "procedure best practice". and it creates bad behaviors. you can add the sleep before throwing the yield exception if you want slow down the retries > ServerCrashProcedure not yielding > - > > Key: HBASE-15591 > URL: https://issues.apache.org/jira/browse/HBASE-15591 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.2.0 >Reporter: Jerry He >Assignee: Jerry He > Attachments: HBASE-15591.patch > > > ServerCrashProcedure is not propagating ProcedureYieldException to the > ProcedureExecutor > One symptom is that while ServerCrashProcedure is waiting for meta to be up > the Procedure WALs get filled up rapidly with all the executions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15591) ServerCrashProcedure not yielding
[ https://issues.apache.org/jira/browse/HBASE-15591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227719#comment-15227719 ] Jerry He commented on HBASE-15591: -- A question for [~mbertozzi] is: the Yielded procedure is put in the queue (although at the back) right away for another execution. Is it reasonable to add a small delay before put it back in the queue? > ServerCrashProcedure not yielding > - > > Key: HBASE-15591 > URL: https://issues.apache.org/jira/browse/HBASE-15591 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.2.0 >Reporter: Jerry He >Assignee: Jerry He > Attachments: HBASE-15591.patch > > > ServerCrashProcedure is not propagating ProcedureYieldException to the > ProcedureExecutor > One symptom is that while ServerCrashProcedure is waiting for meta to be up > the Procedure WALs get filled up rapidly with all the executions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15594) [YCSB] Improvements
[
https://issues.apache.org/jira/browse/HBASE-15594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227711#comment-15227711
]
Duo Zhang commented on HBASE-15594:
---
[~stack] I guess your '48 cpus' are 2 * 12 cores with hyper-threading on? Have
you tried disable the hyper-threading and try again?
I'm not sure whether it is the problem. But hyper-threading does bring some
difficulties to calculate the correct cpu utilization since it is only a
virtual core and can not always perform like a physical core.
Thanks.
> [YCSB] Improvements
> ---
>
> Key: HBASE-15594
> URL: https://issues.apache.org/jira/browse/HBASE-15594
> Project: HBase
> Issue Type: Umbrella
>Reporter: stack
>Priority: Critical
>
> Running YCSB and getting good results is an arcane art. For example, in my
> testing, a few handlers (100) with as many readers as I had CPUs (48), and
> upping connections on clients to same as #cpus made for 2-3x the throughput.
> The above config changes came of lore; which configurations need tweaking is
> not obvious going by their names, there were no indications from the app on
> where/why we were blocked or on which metrics are important to consider. Nor
> was any of this stuff written down in docs.
> Even still, I am stuck trying to make use of all of the machine. I am unable
> to overrun a server though 8 client nodes trying to beat up a single node
> (workloadc, all random-read, with no data returned -p readallfields=false).
> There is also a strange phenomenon where if I add a few machines, rather than
> 3x the YCSB throughput when 3 nodes in cluster, each machine instead is doing
> about 1/3rd.
> This umbrella issue is to host items that improve our defaults and noting how
> to get good numbers running YCSB. In particular, I want to be able to
> saturate a machine.
> Here are the configs I'm currently working with. I've not done the work to
> figure client-side if they are optimal (weird is how big a difference
> client-side changes can make -- need to fix this). On my 48 cpu machine, I
> can do about 370k random reads a second from data totally cached in
> bucketcache. If I short-circuit the user gets so they don't do any work but
> return immediately, I can do 600k ops a second but the CPUs are at 60-70%
> only. I cannot get them to go above this. Working on it.
> {code}
>
>
> hbase.ipc.server.read.threadpool.size
>
> 48
>
>
>
> hbase.regionserver.handler.count
>
> 100
>
>
>
> hbase.client.ipc.pool.size
>
> 100
>
>
>
> hbase.htable.threads.max
>
> 48
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15591) ServerCrashProcedure not yielding
[ https://issues.apache.org/jira/browse/HBASE-15591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227710#comment-15227710 ] Matteo Bertozzi commented on HBASE-15591: - yeah not related, you can go ahead and push for me > ServerCrashProcedure not yielding > - > > Key: HBASE-15591 > URL: https://issues.apache.org/jira/browse/HBASE-15591 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.2.0 >Reporter: Jerry He >Assignee: Jerry He > Attachments: HBASE-15591.patch > > > ServerCrashProcedure is not propagating ProcedureYieldException to the > ProcedureExecutor > One symptom is that while ServerCrashProcedure is waiting for meta to be up > the Procedure WALs get filled up rapidly with all the executions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15591) ServerCrashProcedure not yielding
[ https://issues.apache.org/jira/browse/HBASE-15591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227708#comment-15227708 ] Jerry He commented on HBASE-15591: -- The timeout junit tests don't seem to be related. And they occured in other runs as well: https://issues.apache.org/jira/browse/HBASE-15480?focusedCommentId=15227446=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15227446 org.apache.hadoop.hbase.security.access.TestNamespaceCommands org.apache.hadoop.hbase.snapshot.TestMobFlushSnapshotFromClient > ServerCrashProcedure not yielding > - > > Key: HBASE-15591 > URL: https://issues.apache.org/jira/browse/HBASE-15591 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.2.0 >Reporter: Jerry He >Assignee: Jerry He > Attachments: HBASE-15591.patch > > > ServerCrashProcedure is not propagating ProcedureYieldException to the > ProcedureExecutor > One symptom is that while ServerCrashProcedure is waiting for meta to be up > the Procedure WALs get filled up rapidly with all the executions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15592) Print Procedure WAL content
[ https://issues.apache.org/jira/browse/HBASE-15592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227702#comment-15227702 ] Matteo Bertozzi commented on HBASE-15592: - +1 v2 looks good to me > Print Procedure WAL content > --- > > Key: HBASE-15592 > URL: https://issues.apache.org/jira/browse/HBASE-15592 > Project: HBase > Issue Type: New Feature >Reporter: Jerry He > Attachments: HBASE-15592-1.2-v2.patch, HBASE-15592-master-v2.patch, > HBASE-15592-master.patch > > > Let's have a printer to print the content of Procedure WAL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15592) Print Procedure WAL content
[ https://issues.apache.org/jira/browse/HBASE-15592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He updated HBASE-15592: - Attachment: HBASE-15592-1.2-v2.patch branch 1.x patch has a minor difference: EOF is changed to PROCEDURE_WAL_EOF > Print Procedure WAL content > --- > > Key: HBASE-15592 > URL: https://issues.apache.org/jira/browse/HBASE-15592 > Project: HBase > Issue Type: New Feature >Reporter: Jerry He > Attachments: HBASE-15592-1.2-v2.patch, HBASE-15592-master-v2.patch, > HBASE-15592-master.patch > > > Let's have a printer to print the content of Procedure WAL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15592) Print Procedure WAL content
[ https://issues.apache.org/jira/browse/HBASE-15592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He updated HBASE-15592: - Attachment: HBASE-15592-master-v2.patch Attached a patch that addressed Matteo's comment. Thanks. > Print Procedure WAL content > --- > > Key: HBASE-15592 > URL: https://issues.apache.org/jira/browse/HBASE-15592 > Project: HBase > Issue Type: New Feature >Reporter: Jerry He > Attachments: HBASE-15592-master-v2.patch, HBASE-15592-master.patch > > > Let's have a printer to print the content of Procedure WAL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15594) [YCSB] Improvements
[
https://issues.apache.org/jira/browse/HBASE-15594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227699#comment-15227699
]
stack commented on HBASE-15594:
---
Thank you for chiming in [~carp84] I have a 0.8.0 SNAPSHOT of YCSB that
includes the above fix (sounds like it was a pain to find!):
stack@ve0524:~/YCSB$ git log --oneline|grep 651
b21a3bf Merge pull request #651 from manolama/hbase10_init
Are you able to overrun one of serving RegionServers in your setup? Can you
drive the CPUs to 100%? I am unable. They are always 30-40% idle. Thanks.
> [YCSB] Improvements
> ---
>
> Key: HBASE-15594
> URL: https://issues.apache.org/jira/browse/HBASE-15594
> Project: HBase
> Issue Type: Umbrella
>Reporter: stack
>Priority: Critical
>
> Running YCSB and getting good results is an arcane art. For example, in my
> testing, a few handlers (100) with as many readers as I had CPUs (48), and
> upping connections on clients to same as #cpus made for 2-3x the throughput.
> The above config changes came of lore; which configurations need tweaking is
> not obvious going by their names, there were no indications from the app on
> where/why we were blocked or on which metrics are important to consider. Nor
> was any of this stuff written down in docs.
> Even still, I am stuck trying to make use of all of the machine. I am unable
> to overrun a server though 8 client nodes trying to beat up a single node
> (workloadc, all random-read, with no data returned -p readallfields=false).
> There is also a strange phenomenon where if I add a few machines, rather than
> 3x the YCSB throughput when 3 nodes in cluster, each machine instead is doing
> about 1/3rd.
> This umbrella issue is to host items that improve our defaults and noting how
> to get good numbers running YCSB. In particular, I want to be able to
> saturate a machine.
> Here are the configs I'm currently working with. I've not done the work to
> figure client-side if they are optimal (weird is how big a difference
> client-side changes can make -- need to fix this). On my 48 cpu machine, I
> can do about 370k random reads a second from data totally cached in
> bucketcache. If I short-circuit the user gets so they don't do any work but
> return immediately, I can do 600k ops a second but the CPUs are at 60-70%
> only. I cannot get them to go above this. Working on it.
> {code}
>
>
> hbase.ipc.server.read.threadpool.size
>
> 48
>
>
>
> hbase.regionserver.handler.count
>
> 100
>
>
>
> hbase.client.ipc.pool.size
>
> 100
>
>
>
> hbase.htable.threads.max
>
> 48
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15484) Correct the semantic of batch and partial
[
https://issues.apache.org/jira/browse/HBASE-15484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227691#comment-15227691
]
Hadoop QA commented on HBASE-15484:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 40s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 55s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 4s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 3s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 8s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 1s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 40s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 56s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 56s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 18s
{color} | {color:red} hbase-client: patch generated 30 new + 657 unchanged - 15
fixed = 687 total (was 672) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 56s
{color} | {color:red} hbase-client: patch generated 30 new + 657 unchanged - 15
fixed = 687 total (was 672) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 43s
{color} | {color:red} hbase-server: patch generated 30 new + 657 unchanged - 15
fixed = 687 total (was 672) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 39s
{color} | {color:red} hbase-server: patch generated 30 new + 657 unchanged - 15
fixed = 687 total (was 672) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 27s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 43s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 42s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 56s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 57s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 123m 46s
{color}
[jira] [Commented] (HBASE-15174) Client Public API should not have PB objects in 2.0
[
https://issues.apache.org/jira/browse/HBASE-15174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227685#comment-15227685
]
ramkrishna.s.vasudevan commented on HBASE-15174:
[~enis]
That looks like a good idea. Infact I had not thought about of how to go about
addressing this problem. We would any way next need our patched PB changes to
get in for all the reduction in GCs created. So if this task is a blocker for
that let us first complete it.
> Client Public API should not have PB objects in 2.0
> ---
>
> Key: HBASE-15174
> URL: https://issues.apache.org/jira/browse/HBASE-15174
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 2.0.0
>
>
> Some more cleanup for the parent jira.
> We have leaked some PB structs in Admin (and possible other places).
> We should clean up these API before 2.0.
> Examples include:
> {code}
> AdminProtos.GetRegionInfoResponse.CompactionState getCompactionState(final
> TableName tableName)
> throws IOException;
>
>
> void snapshot(final String snapshotName,
> final TableName tableName,
> HBaseProtos.SnapshotDescription.Type type) throws IOException,
> SnapshotCreationException,
> IllegalArgumentException;
>
> MasterProtos.SnapshotResponse
> takeSnapshotAsync(HBaseProtos.SnapshotDescription snapshot)
> throws IOException, SnapshotCreationException;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15594) [YCSB] Improvements
[
https://issues.apache.org/jira/browse/HBASE-15594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227680#comment-15227680
]
Yu Li commented on HBASE-15594:
---
Yeah, also got a response from YCSB-681. Thanks for the note Sean.
> [YCSB] Improvements
> ---
>
> Key: HBASE-15594
> URL: https://issues.apache.org/jira/browse/HBASE-15594
> Project: HBase
> Issue Type: Umbrella
>Reporter: stack
>Priority: Critical
>
> Running YCSB and getting good results is an arcane art. For example, in my
> testing, a few handlers (100) with as many readers as I had CPUs (48), and
> upping connections on clients to same as #cpus made for 2-3x the throughput.
> The above config changes came of lore; which configurations need tweaking is
> not obvious going by their names, there were no indications from the app on
> where/why we were blocked or on which metrics are important to consider. Nor
> was any of this stuff written down in docs.
> Even still, I am stuck trying to make use of all of the machine. I am unable
> to overrun a server though 8 client nodes trying to beat up a single node
> (workloadc, all random-read, with no data returned -p readallfields=false).
> There is also a strange phenomenon where if I add a few machines, rather than
> 3x the YCSB throughput when 3 nodes in cluster, each machine instead is doing
> about 1/3rd.
> This umbrella issue is to host items that improve our defaults and noting how
> to get good numbers running YCSB. In particular, I want to be able to
> saturate a machine.
> Here are the configs I'm currently working with. I've not done the work to
> figure client-side if they are optimal (weird is how big a difference
> client-side changes can make -- need to fix this). On my 48 cpu machine, I
> can do about 370k random reads a second from data totally cached in
> bucketcache. If I short-circuit the user gets so they don't do any work but
> return immediately, I can do 600k ops a second but the CPUs are at 60-70%
> only. I cannot get them to go above this. Working on it.
> {code}
>
>
> hbase.ipc.server.read.threadpool.size
>
> 48
>
>
>
> hbase.regionserver.handler.count
>
> 100
>
>
>
> hbase.client.ipc.pool.size
>
> 100
>
>
>
> hbase.htable.threads.max
>
> 48
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-15480) Bloom Filter check needs to be more efficient for array
[
https://issues.apache.org/jira/browse/HBASE-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227139#comment-15227139
]
Ted Yu edited comment on HBASE-15480 at 4/6/16 4:08 AM:
{code}
* Check if the specified keys are contained in the bloom filter.
...
+ public BitSet contains(byte[][] key, int[] keyOffset, int[] keyLength,
ByteBuff bloom) {
{code}
Since multiple keys are checked in the method, shouldn't the parameter names be
in plural ?
BTW I don't find the caller for the new method.
Maybe it is used by proprietary code.
was (Author: yuzhih...@gmail.com):
{code}
* Check if the specified keys are contained in the bloom filter.
...
+ public BitSet contains(byte[][] key, int[] keyOffset, int[] keyLength,
ByteBuff bloom) {
{code}
Since multiple keys are checked in the method, shouldn't the parameter names be
in plural ?
BTW I don't find the caller for the new method.
Did I miss something ?
> Bloom Filter check needs to be more efficient for array
> ---
>
> Key: HBASE-15480
> URL: https://issues.apache.org/jira/browse/HBASE-15480
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Affects Versions: 1.0.3
>Reporter: Walter Koetke
>Assignee: Walter Koetke
> Fix For: 1.0.4
>
> Attachments: BloomFilterCheckOneByOne.tiff,
> HBASE-15480-branch-1.0.patch, HBASE-15480.patch
>
>
> It is currently inefficient to do lots of bloom filter checks. Each check has
> overhead like going to the block cache to retrieve the block and recording
> metrics. It would be good to have one bloom filter check api that does a
> bunch of checks without so much block retrieval and metrics updates.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15587) FSTableDescriptors.getDescriptor() logs stack trace erronously
[ https://issues.apache.org/jira/browse/HBASE-15587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227659#comment-15227659 ] Hudson commented on HBASE-15587: SUCCESS: Integrated in HBase-1.2-IT #479 (See [https://builds.apache.org/job/HBase-1.2-IT/479/]) HBASE-15587 FSTableDescriptors.getDescriptor() logs stack trace (enis: rev fcfda192f7a9ab4a7950b2dc21cd861abedf49df) * hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java > FSTableDescriptors.getDescriptor() logs stack trace erronously > -- > > Key: HBASE-15587 > URL: https://issues.apache.org/jira/browse/HBASE-15587 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.3.0, 1.0.4, 0.98.19, 1.4.0, 1.1.5, 1.2.2 > > Attachments: hbase-15587-v1.patch > > > Long time pet-peeve, but it is causing some confusion when looking at the > master logs for unrelated things. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15587) FSTableDescriptors.getDescriptor() logs stack trace erronously
[ https://issues.apache.org/jira/browse/HBASE-15587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227637#comment-15227637 ] Hudson commented on HBASE-15587: SUCCESS: Integrated in HBase-1.3-IT #597 (See [https://builds.apache.org/job/HBase-1.3-IT/597/]) HBASE-15587 FSTableDescriptors.getDescriptor() logs stack trace (enis: rev e6a32141b3e5e893c5b78a2a821c4bbbc6f482ce) * hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java > FSTableDescriptors.getDescriptor() logs stack trace erronously > -- > > Key: HBASE-15587 > URL: https://issues.apache.org/jira/browse/HBASE-15587 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.3.0, 1.0.4, 0.98.19, 1.4.0, 1.1.5, 1.2.2 > > Attachments: hbase-15587-v1.patch > > > Long time pet-peeve, but it is causing some confusion when looking at the > master logs for unrelated things. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15587) FSTableDescriptors.getDescriptor() logs stack trace erronously
[ https://issues.apache.org/jira/browse/HBASE-15587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-15587: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.2.2 1.1.5 1.0.4 Status: Resolved (was: Patch Available) Pushed this to 0.98+. Thanks for reviews. > FSTableDescriptors.getDescriptor() logs stack trace erronously > -- > > Key: HBASE-15587 > URL: https://issues.apache.org/jira/browse/HBASE-15587 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.3.0, 1.0.4, 0.98.19, 1.4.0, 1.1.5, 1.2.2 > > Attachments: hbase-15587-v1.patch > > > Long time pet-peeve, but it is causing some confusion when looking at the > master logs for unrelated things. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227550#comment-15227550 ] Ted Yu commented on HBASE-14123: Ran all backup unit tests which passed - including the TestBackupDelete#testBackupDelete which was disabled in the patch (I enabled it). > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 14123-v14.txt, HBASE-14123-for-7912-v1.patch, > HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15529) Override needBalance in StochasticLoadBalancer
[ https://issues.apache.org/jira/browse/HBASE-15529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227513#comment-15227513 ] Guanghao Zhang commented on HBASE-15529: Ok, I will test it in our own real cluster first. > Override needBalance in StochasticLoadBalancer > -- > > Key: HBASE-15529 > URL: https://issues.apache.org/jira/browse/HBASE-15529 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Minor > Attachments: HBASE-15529-v1.patch, HBASE-15529.patch > > > StochasticLoadBalancer includes cost functions to compute the cost of region > rount, r/w qps, table load, region locality, memstore size, and storefile > size. Every cost function returns a number between 0 and 1 inclusive and the > computed costs are scaled by their respective multipliers. The bigger > multiplier means that the respective cost function have the bigger weight. > But needBalance decide whether to balance only by region count and doesn't > consider r/w qps, locality even you config these cost function with bigger > multiplier. StochasticLoadBalancer should override needBalance and decide > whether to balance by it's configs of cost functions. > Add one new config hbase.master.balancer.stochastic.minCostNeedBalance, > cluster need balance when (total cost / sum multiplier) > minCostNeedBalance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15587) FSTableDescriptors.getDescriptor() logs stack trace erronously
[ https://issues.apache.org/jira/browse/HBASE-15587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227479#comment-15227479 ] Andrew Purtell commented on HBASE-15587: +1 > FSTableDescriptors.getDescriptor() logs stack trace erronously > -- > > Key: HBASE-15587 > URL: https://issues.apache.org/jira/browse/HBASE-15587 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.3.0, 0.98.19, 1.4.0 > > Attachments: hbase-15587-v1.patch > > > Long time pet-peeve, but it is causing some confusion when looking at the > master logs for unrelated things. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15587) FSTableDescriptors.getDescriptor() logs stack trace erronously
[ https://issues.apache.org/jira/browse/HBASE-15587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-15587: --- Fix Version/s: 0.98.19 > FSTableDescriptors.getDescriptor() logs stack trace erronously > -- > > Key: HBASE-15587 > URL: https://issues.apache.org/jira/browse/HBASE-15587 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.3.0, 0.98.19, 1.4.0 > > Attachments: hbase-15587-v1.patch > > > Long time pet-peeve, but it is causing some confusion when looking at the > master logs for unrelated things. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15560) TinyLFU-based BlockCache
[ https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227474#comment-15227474 ] Andrew Purtell commented on HBASE-15560: I don't see a problem making this pronouncement, if and when we take on something 8-only. We're past the end of public updates of Java 7. > TinyLFU-based BlockCache > > > Key: HBASE-15560 > URL: https://issues.apache.org/jira/browse/HBASE-15560 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Reporter: Ben Manes > Attachments: tinylfu.patch > > > LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and > recency of the working set. It achieves concurrency by using an O( n ) > background thread to prioritize the entries and evict. Accessing an entry is > O(1) by a hash table lookup, recording its logical access time, and setting a > frequency flag. A write is performed in O(1) time by updating the hash table > and triggering an async eviction thread. This provides ideal concurrency and > minimizes the latencies by penalizing the thread instead of the caller. > However the policy does not age the frequencies and may not be resilient to > various workload patterns. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > Concurrency is achieved by buffering and replaying the operations, similar to > a write-ahead log. A read is recorded into a striped ring buffer and writes > to a queue. The operations are applied in batches under a try-lock by an > asynchronous thread, thereby track the usage pattern without incurring high > latencies > ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]). > In YCSB benchmarks the results were inconclusive. For a large cache (99% hit > rates) the two caches have near identical throughput and latencies with > LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a > 1-4% hit rate improvement and therefore lower latencies. The lack luster > result is because a synthetic Zipfian distribution is used, which SLRU > performs optimally. In a more varied, real-world workload we'd expect to see > improvements by being able to make smarter predictions. > The provided patch implements BlockCache using the > [Caffeine|https://github.com/ben-manes/caffeine] caching library (see > HighScalability > [article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]). > Edward Bortnikov and Eshcar Hillel have graciously provided guidance for > evaluating this patch ([github > branch|https://github.com/ben-manes/hbase/tree/tinylfu]). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15537) Add multi WAL support for AsyncFSWAL
[
https://issues.apache.org/jira/browse/HBASE-15537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227472#comment-15227472
]
Duo Zhang commented on HBASE-15537:
---
Fine. Will commit shortly if no objection.
Thanks.
> Add multi WAL support for AsyncFSWAL
>
>
> Key: HBASE-15537
> URL: https://issues.apache.org/jira/browse/HBASE-15537
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15537-v3.patch, HBASE-15537-v4.patch,
> HBASE-15537-v5.patch, HBASE-15537.patch, HBASE-15537_v2.patch
>
>
> The multi WAL should not be bound with {{FSHLog}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15537) Add multi WAL support for AsyncFSWAL
[
https://issues.apache.org/jira/browse/HBASE-15537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-15537:
--
Release Note:
Add the delegate config for multiwal back. Now you can use
'hbase.wal.regiongrouping.delegate.provider' to specify the wal provider you
want to use for multiwal. For example:
{code}
hbase.wal.regiongrouping.delegate.provider
asyncfs
{code}
And the default value is filesystem which is the alias of DefaultWALProvider,
i.e., the FSHLog.
was:
Add the delegate config for multiwal back. Now you can use
'hbase.wal.regiongrouping.delegate.provider' to specify the wal provider you
want to use for multiwal. For example:
{code}
hbase.wal.regiongrouping.delegate.provider
asyncfs
{code}
And the default value is filesystem which is the alias of DefaultWALProvider,
i.e., the FSHLog.
> Add multi WAL support for AsyncFSWAL
>
>
> Key: HBASE-15537
> URL: https://issues.apache.org/jira/browse/HBASE-15537
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15537-v3.patch, HBASE-15537-v4.patch,
> HBASE-15537-v5.patch, HBASE-15537.patch, HBASE-15537_v2.patch
>
>
> The multi WAL should not be bound with {{FSHLog}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15407) Add SASL support for fan out OutputStream
[ https://issues.apache.org/jira/browse/HBASE-15407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227457#comment-15227457 ] Duo Zhang commented on HBASE-15407: --- [~stack] Any progress here? Thanks. > Add SASL support for fan out OutputStream > - > > Key: HBASE-15407 > URL: https://issues.apache.org/jira/browse/HBASE-15407 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang > Attachments: HBASE-15407-v1.patch, HBASE-15407-v2.patch, > HBASE-15407-v3.patch, HBASE-15407-v4.patch, HBASE-15407-v5.patch, > HBASE-15407.patch > > > Otherwise we can not use it in secure environment. > Should be a netty handler, but see > https://github.com/netty/netty/issues/1966 > I do not think it will be available in the near future, so we need to do it > by ourselves. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15480) Bloom Filter check needs to be more efficient for array
[
https://issues.apache.org/jira/browse/HBASE-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227446#comment-15227446
]
Hadoop QA commented on HBASE-15480:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 42s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 21s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 34s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 5m 46s
{color} | {color:red} hbase-server: patch generated 4 new + 2 unchanged - 0
fixed = 6 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
39m 43s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 194m 41s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 274m 26s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
| | hadoop.hbase.master.balancer.TestStochasticLoadBalancer2 |
| Timed out junit tests |
org.apache.hadoop.hbase.security.access.TestNamespaceCommands |
| | org.apache.hadoop.hbase.snapshot.TestMobFlushSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12797151/HBASE-15480.patch |
| JIRA Issue | HBASE-15480 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux penates.apache.org 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT
Wed Sep 3
[jira] [Commented] (HBASE-15505) ReplicationPeerConfig should be builder-style
[
https://issues.apache.org/jira/browse/HBASE-15505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227384#comment-15227384
]
Hudson commented on HBASE-15505:
FAILURE: Integrated in HBase-Trunk_matrix #826 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/826/])
HBASE-15505 ReplicationPeerConfig should be builder-style (Gabor Liptak) (enis:
rev 7e399883f62fd37e5215ce3a456a917e690c921c)
*
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestUnmodifyableHTableDescriptor.java
*
hbase-client/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationPeerConfig.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/UnmodifyableHTableDescriptor.java
* hbase-client/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaFilter.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeerConfig.java
> ReplicationPeerConfig should be builder-style
> -
>
> Key: HBASE-15505
> URL: https://issues.apache.org/jira/browse/HBASE-15505
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Gabor Liptak
> Labels: beginner, beginners
> Fix For: 2.0.0
>
> Attachments: HBASE-15505.1.patch, HBASE-15505.2.patch,
> HBASE-15505.3.patch, HBASE-15505.4.patch
>
>
> Similar to HTD, HCD, {{ReplicationPeerConfig}} should be builder-style:
> {code}
> ReplicationPeerConfig().setFoo(foo).setBar(bar);
> {code}
> See {{TestHTableDescriptor.testClassMethodsAreBuilderStyle()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15369) Handle NPE in region.jsp
[
https://issues.apache.org/jira/browse/HBASE-15369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227320#comment-15227320
]
Ted Yu commented on HBASE-15369:
{code}
41if(region == null) {
42 displayName= ESAPI.encoder().encodeForHTML(regionName) + " does
not exist";
43}else {
{code}
Please insert space between if and '(', between right curly and else.
Please replace tab characters.
> Handle NPE in region.jsp
>
>
> Key: HBASE-15369
> URL: https://issues.apache.org/jira/browse/HBASE-15369
> Project: HBase
> Issue Type: Sub-task
> Components: UI
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-15369_v0.patch
>
>
> If empty or non-existing region name parameter is passed to "region.jsp
> ?name=" it will cause 500 NPE error. It is not big deal but looks ugly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15572) Adding optional timestamp semantics to HBase-Spark
[
https://issues.apache.org/jira/browse/HBASE-15572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227314#comment-15227314
]
Hadoop QA commented on HBASE-15572:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 20s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 39s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 6s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 56s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} scaladoc {color} | {color:red} 0m 8s
{color} | {color:red} root in master failed. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 3s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 9m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 44s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 8m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 56s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 38s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 17s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} scaladoc {color} | {color:red} 0m 9s
{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} scaladoc {color} | {color:red} 0m 7s
{color} | {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 22s
{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 19s
{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 18s
{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 16s
{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 151m 45s
{color} | {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
19s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 272m 10s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
[jira] [Commented] (HBASE-15572) Adding optional timestamp semantics to HBase-Spark
[ https://issues.apache.org/jira/browse/HBASE-15572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227292#comment-15227292 ] Sean Busbey commented on HBASE-15572: - yep. will take a look this evening. > Adding optional timestamp semantics to HBase-Spark > -- > > Key: HBASE-15572 > URL: https://issues.apache.org/jira/browse/HBASE-15572 > Project: HBase > Issue Type: New Feature > Components: spark >Reporter: Weiqing Yang >Assignee: Weiqing Yang > Fix For: 2.0.0 > > Attachments: HBASE-15572-1.patch, HBASE-15572-2.patch, > HBASE-15572-3.patch, HBASE-15572-4.patch, HBASE-15572-5.patch, > HBASE-15572-6.patch, HBASE-15572-7.patch, HBASE-15572-8.patch > > > Right now the timestamp is always latest. With this patch, users can select > timestamps they want. > In this patch, 4 parameters, "timestamp", "minTimestamp", "maxiTimestamp" and > "maxVersions" are added to HBaseSparkConf. Users can select a timestamp, they > can also select a time range with minimum timestamp and maximum timestamp. A > new test for selecting records with different timestamps is added. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15594) [YCSB] Improvements
[
https://issues.apache.org/jira/browse/HBASE-15594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227283#comment-15227283
]
Sean Busbey commented on HBASE-15594:
-
The number of connections bug was fixed in
[YCSB-651|https://github.com/brianfrankcooper/YCSB/pull/651], which will be
included in the 0.8.0 release, which is in candidates now.
> [YCSB] Improvements
> ---
>
> Key: HBASE-15594
> URL: https://issues.apache.org/jira/browse/HBASE-15594
> Project: HBase
> Issue Type: Umbrella
>Reporter: stack
>Priority: Critical
>
> Running YCSB and getting good results is an arcane art. For example, in my
> testing, a few handlers (100) with as many readers as I had CPUs (48), and
> upping connections on clients to same as #cpus made for 2-3x the throughput.
> The above config changes came of lore; which configurations need tweaking is
> not obvious going by their names, there were no indications from the app on
> where/why we were blocked or on which metrics are important to consider. Nor
> was any of this stuff written down in docs.
> Even still, I am stuck trying to make use of all of the machine. I am unable
> to overrun a server though 8 client nodes trying to beat up a single node
> (workloadc, all random-read, with no data returned -p readallfields=false).
> There is also a strange phenomenon where if I add a few machines, rather than
> 3x the YCSB throughput when 3 nodes in cluster, each machine instead is doing
> about 1/3rd.
> This umbrella issue is to host items that improve our defaults and noting how
> to get good numbers running YCSB. In particular, I want to be able to
> saturate a machine.
> Here are the configs I'm currently working with. I've not done the work to
> figure client-side if they are optimal (weird is how big a difference
> client-side changes can make -- need to fix this). On my 48 cpu machine, I
> can do about 370k random reads a second from data totally cached in
> bucketcache. If I short-circuit the user gets so they don't do any work but
> return immediately, I can do 600k ops a second but the CPUs are at 60-70%
> only. I cannot get them to go above this. Working on it.
> {code}
>
>
> hbase.ipc.server.read.threadpool.size
>
> 48
>
>
>
> hbase.regionserver.handler.count
>
> 100
>
>
>
> hbase.client.ipc.pool.size
>
> 100
>
>
>
> hbase.htable.threads.max
>
> 48
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15587) FSTableDescriptors.getDescriptor() logs stack trace erronously
[ https://issues.apache.org/jira/browse/HBASE-15587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227282#comment-15227282 ] Matteo Bertozzi commented on HBASE-15587: - +1 > FSTableDescriptors.getDescriptor() logs stack trace erronously > -- > > Key: HBASE-15587 > URL: https://issues.apache.org/jira/browse/HBASE-15587 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: hbase-15587-v1.patch > > > Long time pet-peeve, but it is causing some confusion when looking at the > master logs for unrelated things. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15507) Online modification of enabled ReplicationPeerConfig
[ https://issues.apache.org/jira/browse/HBASE-15507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Jacoby updated HBASE-15507: Labels: Replication shell (was: ) Fix Version/s: 2.0.0 Release Note: Added update_peer to the HBase shell and ReplicationAdmin, and provided a callback for custom replication endpoints to be notified of changes to their configuration and peer data Status: Patch Available (was: In Progress) Added update_peer to the HBase shell and ReplicationAdmin, and provided a callback for custom replication endpoints to be notified of changes to their configuration and peer data. > Online modification of enabled ReplicationPeerConfig > > > Key: HBASE-15507 > URL: https://issues.apache.org/jira/browse/HBASE-15507 > Project: HBase > Issue Type: Improvement > Components: Replication >Affects Versions: 1.2.0, 2.0.0, 1.3.0 >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby > Labels: Replication, shell > Fix For: 2.0.0 > > Attachments: HBASE-15507.patch > > > It's currently possible to update the table CFs for a replication peer while > it's running, but not the peer configuration or data maps introduced as part > of custom replication endpoints in HBASE-11367. > This means that if you have a custom endpoint that depends on some > configuration parameter, you have to remove the peer and recreate it in order > to change the param. In some use cases that's not possible without risking > data loss. > HBASE-11393, which will consolidate tableCFs in the same znode as the rest of > ReplicationPeerConfig, may help here, but with or without it it still seems > like there needs to be further work to add update config/data API support to > ReplicationAdmin and a callback mechanism to notify the endpoints of config > parameter changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15507) Online modification of enabled ReplicationPeerConfig
[ https://issues.apache.org/jira/browse/HBASE-15507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Jacoby updated HBASE-15507: Attachment: HBASE-15507.patch > Online modification of enabled ReplicationPeerConfig > > > Key: HBASE-15507 > URL: https://issues.apache.org/jira/browse/HBASE-15507 > Project: HBase > Issue Type: Improvement > Components: Replication >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby > Attachments: HBASE-15507.patch > > > It's currently possible to update the table CFs for a replication peer while > it's running, but not the peer configuration or data maps introduced as part > of custom replication endpoints in HBASE-11367. > This means that if you have a custom endpoint that depends on some > configuration parameter, you have to remove the peer and recreate it in order > to change the param. In some use cases that's not possible without risking > data loss. > HBASE-11393, which will consolidate tableCFs in the same znode as the rest of > ReplicationPeerConfig, may help here, but with or without it it still seems > like there needs to be further work to add update config/data API support to > ReplicationAdmin and a callback mechanism to notify the endpoints of config > parameter changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15529) Override needBalance in StochasticLoadBalancer
[ https://issues.apache.org/jira/browse/HBASE-15529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227230#comment-15227230 ] Ted Yu commented on HBASE-15529: Also, please turn on region replica for some table(s) in the cluster. Make sure they're balanced. > Override needBalance in StochasticLoadBalancer > -- > > Key: HBASE-15529 > URL: https://issues.apache.org/jira/browse/HBASE-15529 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Minor > Attachments: HBASE-15529-v1.patch, HBASE-15529.patch > > > StochasticLoadBalancer includes cost functions to compute the cost of region > rount, r/w qps, table load, region locality, memstore size, and storefile > size. Every cost function returns a number between 0 and 1 inclusive and the > computed costs are scaled by their respective multipliers. The bigger > multiplier means that the respective cost function have the bigger weight. > But needBalance decide whether to balance only by region count and doesn't > consider r/w qps, locality even you config these cost function with bigger > multiplier. StochasticLoadBalancer should override needBalance and decide > whether to balance by it's configs of cost functions. > Add one new config hbase.master.balancer.stochastic.minCostNeedBalance, > cluster need balance when (total cost / sum multiplier) > minCostNeedBalance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15400) Use DateTieredCompactor for Date Tiered Compaction
[ https://issues.apache.org/jira/browse/HBASE-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227144#comment-15227144 ] Clara Xiong commented on HBASE-15400: - Test failure is not related and passes locally. [~Apache9] [~enis] Do you have more concerns? If not, can we get this into trunk? Once this is approved, I will make the branch-1 and 98 patches. We should get it with HBASE-15389 into 1.3 too. > Use DateTieredCompactor for Date Tiered Compaction > -- > > Key: HBASE-15400 > URL: https://issues.apache.org/jira/browse/HBASE-15400 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Clara Xiong >Assignee: Clara Xiong > Fix For: 2.0.0 > > Attachments: HBASE-15400-15389-v12.patch, HBASE-15400-v1.pa, > HBASE-15400-v3-v3.patch, HBASE-15400-v3-v4.patch, HBASE-15400-v3-v5.patch, > HBASE-15400-v3.patch, HBASE-15400.patch > > > When we compact, we can output multiple files along the current window > boundaries. There are two use cases: > 1. Major compaction: We want to output date tiered store files with data > older than max age archived in trunks of the window size on the higher tier. > Once a window is old enough, we don't combine the windows to promote to the > next tier any further. So files in these windows retain the same timespan as > they were minor-compacted last time, which is the window size of the highest > tier. Major compaction will touch these files and we want to maintain the > same layout. This way, TTL and archiving will be simpler and more efficient. > 2. Bulk load files and the old file generated by major compaction before > upgrading to DTCP. > Pros: > 1. Restore locality, process versioning, updates and deletes while > maintaining the tiered layout. > 2. The best way to fix a skewed layout. > > This work is based on a prototype of DateTieredCompactor from HBASE-15389 and > focused on the part to meet needs for these two use cases while supporting > others. I have to call out a few design decisions: > 1. We only want to output the files along all windows for major compaction. > And we want to output multiple files older than max age in the trunks of the > maximum tier window size determined by base window size, windows per tier and > max age. > 2. For minor compaction, we don't want to output too many files, which will > remain around because of current restriction of contiguous compaction by seq > id. I will only output two files if all the files in the windows are being > combined, one for the data within window and the other for the out-of-window > tail. If there is any file in the window excluded from compaction, only one > file will be output from compaction. When the windows are promoted, the > situation of out of order data will gradually improve. For the incoming > window, we need to accommodate the case with user-specified future data. > 3. We have to pass the boundaries with the list of store file as a complete > time snapshot instead of two separate calls because window layout is > determined by the time the computation is called. So we will need new type of > compaction request. > 4. Since we will assign the same seq id for all output files, we need to sort > by maxTimestamp subsequently. Right now all compaction policy gets the files > sorted for StoreFileManager which sorts by seq id and other criteria. I will > use this order for DTCP only, to avoid impacting other compaction policies. > 5. We need some cleanup of current design of StoreEngine and CompactionPolicy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15480) Bloom Filter check needs to be more efficient for array
[
https://issues.apache.org/jira/browse/HBASE-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227139#comment-15227139
]
Ted Yu commented on HBASE-15480:
{code}
* Check if the specified keys are contained in the bloom filter.
...
+ public BitSet contains(byte[][] key, int[] keyOffset, int[] keyLength,
ByteBuff bloom) {
{code}
Since multiple keys are checked in the method, shouldn't the parameter names be
in plural ?
BTW I don't find the caller for the new method.
Did I miss something ?
> Bloom Filter check needs to be more efficient for array
> ---
>
> Key: HBASE-15480
> URL: https://issues.apache.org/jira/browse/HBASE-15480
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Affects Versions: 1.0.3
>Reporter: Walter Koetke
>Assignee: Walter Koetke
> Fix For: 1.0.4
>
> Attachments: BloomFilterCheckOneByOne.tiff,
> HBASE-15480-branch-1.0.patch, HBASE-15480.patch
>
>
> It is currently inefficient to do lots of bloom filter checks. Each check has
> overhead like going to the block cache to retrieve the block and recording
> metrics. It would be good to have one bloom filter check api that does a
> bunch of checks without so much block retrieval and metrics updates.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15519) Add per-user metrics
[ https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-15519: -- Affects Version/s: 1.2.0 Status: Patch Available (was: Open) > Add per-user metrics > - > > Key: HBASE-15519 > URL: https://issues.apache.org/jira/browse/HBASE-15519 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 1.2.0 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Attachments: hbase-15519_v0.patch, hbase-15519_v1.patch > > > Per-user metrics will be useful in multi-tenant cases where we can emit > number of requests, operations, num RPCs etc at the per-user aggregate level > per regionserver. We currently have throttles per user, but no way to monitor > resource usage per-user. > Looking at these metrics, operators can adjust throttles, do capacity > planning, etc per-user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15400) Use DateTieredCompactor for Date Tiered Compaction
[
https://issues.apache.org/jira/browse/HBASE-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227053#comment-15227053
]
Hadoop QA commented on HBASE-15400:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 36s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 39s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 35s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
59s {color} | {color:green} hbase-server: patch generated 0 new + 110 unchanged
- 28 fixed = 110 total (was 138) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
49m 56s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 121m 53s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 2m
29s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 213m 7s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.replication.TestReplicationKillSlaveRS |
| | hadoop.hbase.master.balancer.TestStochasticLoadBalancer2 |
| | hadoop.hbase.regionserver.TestEncryptionKeyRotation |
| Timed out junit tests |
org.apache.hadoop.hbase.replication.TestReplicationSmallTests |
| | org.apache.hadoop.hbase.replication.TestReplicationKillMasterRS |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12797111/HBASE-15400-v3-v5.patch
|
| JIRA Issue | HBASE-15400 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
|
[jira] [Commented] (HBASE-15594) [YCSB] Improvements
[
https://issues.apache.org/jira/browse/HBASE-15594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227017#comment-15227017
]
stack commented on HBASE-15594:
---
Thanks for pointer [~busbey] ... will do both (one point at other).
> [YCSB] Improvements
> ---
>
> Key: HBASE-15594
> URL: https://issues.apache.org/jira/browse/HBASE-15594
> Project: HBase
> Issue Type: Umbrella
>Reporter: stack
>Priority: Critical
>
> Running YCSB and getting good results is an arcane art. For example, in my
> testing, a few handlers (100) with as many readers as I had CPUs (48), and
> upping connections on clients to same as #cpus made for 2-3x the throughput.
> The above config changes came of lore; which configurations need tweaking is
> not obvious going by their names, there were no indications from the app on
> where/why we were blocked or on which metrics are important to consider. Nor
> was any of this stuff written down in docs.
> Even still, I am stuck trying to make use of all of the machine. I am unable
> to overrun a server though 8 client nodes trying to beat up a single node
> (workloadc, all random-read, with no data returned -p readallfields=false).
> There is also a strange phenomenon where if I add a few machines, rather than
> 3x the YCSB throughput when 3 nodes in cluster, each machine instead is doing
> about 1/3rd.
> This umbrella issue is to host items that improve our defaults and noting how
> to get good numbers running YCSB. In particular, I want to be able to
> saturate a machine.
> Here are the configs I'm currently working with. I've not done the work to
> figure client-side if they are optimal (weird is how big a difference
> client-side changes can make -- need to fix this). On my 48 cpu machine, I
> can do about 370k random reads a second from data totally cached in
> bucketcache. If I short-circuit the user gets so they don't do any work but
> return immediately, I can do 600k ops a second but the CPUs are at 60-70%
> only. I cannot get them to go above this. Working on it.
> {code}
>
>
> hbase.ipc.server.read.threadpool.size
>
> 48
>
>
>
> hbase.regionserver.handler.count
>
> 100
>
>
>
> hbase.client.ipc.pool.size
>
> 100
>
>
>
> hbase.htable.threads.max
>
> 48
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15594) [YCSB] Improvements
[
https://issues.apache.org/jira/browse/HBASE-15594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227019#comment-15227019
]
stack commented on HBASE-15594:
---
I'm trying to hit 100% but am unable.
> [YCSB] Improvements
> ---
>
> Key: HBASE-15594
> URL: https://issues.apache.org/jira/browse/HBASE-15594
> Project: HBase
> Issue Type: Umbrella
>Reporter: stack
>Priority: Critical
>
> Running YCSB and getting good results is an arcane art. For example, in my
> testing, a few handlers (100) with as many readers as I had CPUs (48), and
> upping connections on clients to same as #cpus made for 2-3x the throughput.
> The above config changes came of lore; which configurations need tweaking is
> not obvious going by their names, there were no indications from the app on
> where/why we were blocked or on which metrics are important to consider. Nor
> was any of this stuff written down in docs.
> Even still, I am stuck trying to make use of all of the machine. I am unable
> to overrun a server though 8 client nodes trying to beat up a single node
> (workloadc, all random-read, with no data returned -p readallfields=false).
> There is also a strange phenomenon where if I add a few machines, rather than
> 3x the YCSB throughput when 3 nodes in cluster, each machine instead is doing
> about 1/3rd.
> This umbrella issue is to host items that improve our defaults and noting how
> to get good numbers running YCSB. In particular, I want to be able to
> saturate a machine.
> Here are the configs I'm currently working with. I've not done the work to
> figure client-side if they are optimal (weird is how big a difference
> client-side changes can make -- need to fix this). On my 48 cpu machine, I
> can do about 370k random reads a second from data totally cached in
> bucketcache. If I short-circuit the user gets so they don't do any work but
> return immediately, I can do 600k ops a second but the CPUs are at 60-70%
> only. I cannot get them to go above this. Working on it.
> {code}
>
>
> hbase.ipc.server.read.threadpool.size
>
> 48
>
>
>
> hbase.regionserver.handler.count
>
> 100
>
>
>
> hbase.client.ipc.pool.size
>
> 100
>
>
>
> hbase.htable.threads.max
>
> 48
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15594) [YCSB] Improvements
[
https://issues.apache.org/jira/browse/HBASE-15594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226971#comment-15226971
]
Yu Li commented on HBASE-15594:
---
Also raised an [issue|https://github.com/brianfrankcooper/YCSB/issues/681] in
YCSB, JFYI.
> [YCSB] Improvements
> ---
>
> Key: HBASE-15594
> URL: https://issues.apache.org/jira/browse/HBASE-15594
> Project: HBase
> Issue Type: Umbrella
>Reporter: stack
>Priority: Critical
>
> Running YCSB and getting good results is an arcane art. For example, in my
> testing, a few handlers (100) with as many readers as I had CPUs (48), and
> upping connections on clients to same as #cpus made for 2-3x the throughput.
> The above config changes came of lore; which configurations need tweaking is
> not obvious going by their names, there were no indications from the app on
> where/why we were blocked or on which metrics are important to consider. Nor
> was any of this stuff written down in docs.
> Even still, I am stuck trying to make use of all of the machine. I am unable
> to overrun a server though 8 client nodes trying to beat up a single node
> (workloadc, all random-read, with no data returned -p readallfields=false).
> There is also a strange phenomenon where if I add a few machines, rather than
> 3x the YCSB throughput when 3 nodes in cluster, each machine instead is doing
> about 1/3rd.
> This umbrella issue is to host items that improve our defaults and noting how
> to get good numbers running YCSB. In particular, I want to be able to
> saturate a machine.
> Here are the configs I'm currently working with. I've not done the work to
> figure client-side if they are optimal (weird is how big a difference
> client-side changes can make -- need to fix this). On my 48 cpu machine, I
> can do about 370k random reads a second from data totally cached in
> bucketcache. If I short-circuit the user gets so they don't do any work but
> return immediately, I can do 600k ops a second but the CPUs are at 60-70%
> only. I cannot get them to go above this. Working on it.
> {code}
>
>
> hbase.ipc.server.read.threadpool.size
>
> 48
>
>
>
> hbase.regionserver.handler.count
>
> 100
>
>
>
> hbase.client.ipc.pool.size
>
> 100
>
>
>
> hbase.htable.threads.max
>
> 48
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15480) Bloom Filter check needs to be more efficient for array
[ https://issues.apache.org/jira/browse/HBASE-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Koetke updated HBASE-15480: -- Status: Patch Available (was: In Progress) > Bloom Filter check needs to be more efficient for array > --- > > Key: HBASE-15480 > URL: https://issues.apache.org/jira/browse/HBASE-15480 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 1.0.3 >Reporter: Walter Koetke >Assignee: Walter Koetke > Fix For: 1.0.4 > > Attachments: BloomFilterCheckOneByOne.tiff, > HBASE-15480-branch-1.0.patch, HBASE-15480.patch > > > It is currently inefficient to do lots of bloom filter checks. Each check has > overhead like going to the block cache to retrieve the block and recording > metrics. It would be good to have one bloom filter check api that does a > bunch of checks without so much block retrieval and metrics updates. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15480) Bloom Filter check needs to be more efficient for array
[ https://issues.apache.org/jira/browse/HBASE-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Koetke updated HBASE-15480: -- Attachment: HBASE-15480.patch Attaching patch file for master > Bloom Filter check needs to be more efficient for array > --- > > Key: HBASE-15480 > URL: https://issues.apache.org/jira/browse/HBASE-15480 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 1.0.3 >Reporter: Walter Koetke >Assignee: Walter Koetke > Fix For: 1.0.4 > > Attachments: BloomFilterCheckOneByOne.tiff, > HBASE-15480-branch-1.0.patch, HBASE-15480.patch > > > It is currently inefficient to do lots of bloom filter checks. Each check has > overhead like going to the block cache to retrieve the block and recording > metrics. It would be good to have one bloom filter check api that does a > bunch of checks without so much block retrieval and metrics updates. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15595) Document new metrics improvements
[ https://issues.apache.org/jira/browse/HBASE-15595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226958#comment-15226958 ] Yu Li commented on HBASE-15595: --- Thanks [~enis] for opening this JIRA and following up this thread! Just back to finish the job here. :-) > Document new metrics improvements > - > > Key: HBASE-15595 > URL: https://issues.apache.org/jira/browse/HBASE-15595 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar > > We should document the improvements from the parent jira when we are done. > Per-table, per-user metrics, dfs metrics, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
[
https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226923#comment-15226923
]
Hadoop QA commented on HBASE-15525:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 31s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 13s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 13s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 41s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 22s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 1s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 14s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 10s
{color} | {color:red} hbase-common: patch generated 12 new + 18 unchanged - 0
fixed = 30 total (was 18) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 4s
{color} | {color:red} hbase-common: patch generated 12 new + 18 unchanged - 0
fixed = 30 total (was 18) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 4s
{color} | {color:red} hbase-client: patch generated 12 new + 18 unchanged - 0
fixed = 30 total (was 18) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 3s
{color} | {color:red} hbase-client: patch generated 12 new + 18 unchanged - 0
fixed = 30 total (was 18) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 2s
{color} | {color:red} hbase-server: patch generated 12 new + 18 unchanged - 0
fixed = 30 total (was 18) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 3s
{color} | {color:red} hbase-server: patch generated 12 new + 18 unchanged - 0
fixed = 30 total (was 18) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
13s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 8 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 49s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m 4s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 4m 52s
{color} | {color:red} hbase-common-jdk1.8.0 with JDK v1.8.0 generated 6 new + 0
unchanged - 0 fixed = 6 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} |
[jira] [Commented] (HBASE-15594) [YCSB] Improvements
[
https://issues.apache.org/jira/browse/HBASE-15594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226919#comment-15226919
]
Yu Li commented on HBASE-15594:
---
Which version of YCSB you're running sir [~stack]? We happen to do benchmarking
these days comparing 1.1.2 (with miscellaneous fixes on increment as well as
reader/writer rowlock backported) and 0.98.12 using ycsb-0.7.0, and it turns
out the pure read(get) performance declined a lot in our 1.1.2. After days of
debugging (actually just located the root cause few hours ago), we found that
there's a *small but fatal* bug in {{HBaseClient10}} that it initializes one
connection per thread. We could find below codes in {{HBaseClient10#init}},
line 135:
{code}
THREAD_COUNT.getAndIncrement();
synchronized(THREAD_COUNT) {
connection = ConnectionFactory.createConnection(config);
}
{code}
After fixing it in below way, the performance recovered:
{code}
THREAD_COUNT.getAndIncrement();
synchronized(THREAD_COUNT) {
if(connection == null) connection =
ConnectionFactory.createConnection(config);
}
{code}
We were using 4 physical nodes as client, each node ran 8 YCSB process, each
process launched 100 threads. We loaded 100GB data into 3 RS cluster, and then
ran each client to do random get for 30 min, and w/o the fix 1.1.2 performance
is ~20% lower than 0.98.12, which could be reproduced steadily. Hope this
information could help.
> [YCSB] Improvements
> ---
>
> Key: HBASE-15594
> URL: https://issues.apache.org/jira/browse/HBASE-15594
> Project: HBase
> Issue Type: Umbrella
>Reporter: stack
>Priority: Critical
>
> Running YCSB and getting good results is an arcane art. For example, in my
> testing, a few handlers (100) with as many readers as I had CPUs (48), and
> upping connections on clients to same as #cpus made for 2-3x the throughput.
> The above config changes came of lore; which configurations need tweaking is
> not obvious going by their names, there were no indications from the app on
> where/why we were blocked or on which metrics are important to consider. Nor
> was any of this stuff written down in docs.
> Even still, I am stuck trying to make use of all of the machine. I am unable
> to overrun a server though 8 client nodes trying to beat up a single node
> (workloadc, all random-read, with no data returned -p readallfields=false).
> There is also a strange phenomenon where if I add a few machines, rather than
> 3x the YCSB throughput when 3 nodes in cluster, each machine instead is doing
> about 1/3rd.
> This umbrella issue is to host items that improve our defaults and noting how
> to get good numbers running YCSB. In particular, I want to be able to
> saturate a machine.
> Here are the configs I'm currently working with. I've not done the work to
> figure client-side if they are optimal (weird is how big a difference
> client-side changes can make -- need to fix this). On my 48 cpu machine, I
> can do about 370k random reads a second from data totally cached in
> bucketcache. If I short-circuit the user gets so they don't do any work but
> return immediately, I can do 600k ops a second but the CPUs are at 60-70%
> only. I cannot get them to go above this. Working on it.
> {code}
>
>
> hbase.ipc.server.read.threadpool.size
>
> 48
>
>
>
> hbase.regionserver.handler.count
>
> 100
>
>
>
> hbase.client.ipc.pool.size
>
> 100
>
>
>
> hbase.htable.threads.max
>
> 48
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15505) ReplicationPeerConfig should be builder-style
[
https://issues.apache.org/jira/browse/HBASE-15505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-15505:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Release Note: (was: HBASE-15505 ReplicationPeerConfig should be
builder-style)
Status: Resolved (was: Patch Available)
I've committed this. Thanks Gabor for the patch.
> ReplicationPeerConfig should be builder-style
> -
>
> Key: HBASE-15505
> URL: https://issues.apache.org/jira/browse/HBASE-15505
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Gabor Liptak
> Labels: beginner, beginners
> Fix For: 2.0.0
>
> Attachments: HBASE-15505.1.patch, HBASE-15505.2.patch,
> HBASE-15505.3.patch, HBASE-15505.4.patch
>
>
> Similar to HTD, HCD, {{ReplicationPeerConfig}} should be builder-style:
> {code}
> ReplicationPeerConfig().setFoo(foo).setBar(bar);
> {code}
> See {{TestHTableDescriptor.testClassMethodsAreBuilderStyle()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15519) Add per-user metrics
[
https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-15519:
--
Attachment: hbase-15519_v1.patch
Here is v1 patch, ready for reviews.
To measure any possible impact, I've ran a YCSB setup on a single node.
workloadc, all random reads, all of the data is in block cache, 3M records and
3M operations. Client with 100 threads running in a different node. 24 CPUs,
16GB heap, 105 handlers. Code is branch-1 head.
Results without patch:
{code}
Run1:
[OVERALL], RunTime(ms), 106555.0
[OVERALL], Throughput(ops/sec), 28154.47421519403
[READ], Operations, 300.0
[READ], AverageLatency(us), 2684.28608367
[READ], MinLatency(us), 206.0
[READ], MaxLatency(us), 334079.0
[READ], 95thPercentileLatency(us), 4455.0
[READ], 99thPercentileLatency(us), 5699.0
[READ], Return=OK, 300
Run2:
[OVERALL], RunTime(ms), 107010.0
[OVERALL], Throughput(ops/sec), 28034.763106251754
[READ], Operations, 300.0
[READ], AverageLatency(us), 2689.838134
[READ], MinLatency(us), 200.0
[READ], MaxLatency(us), 245759.0
[READ], 95thPercentileLatency(us), 4299.0
[READ], 99thPercentileLatency(us), 5307.0
[READ], Return=OK, 300
Run3:
[OVERALL], RunTime(ms), 109902.0
[OVERALL], Throughput(ops/sec), 27297.046459573074
[READ], Operations, 300.0
[READ], AverageLatency(us), 2774.884157334
[READ], MinLatency(us), 193.0
[READ], MaxLatency(us), 265471.0
[READ], 95thPercentileLatency(us), 4427.0
[READ], 99thPercentileLatency(us), 5791.0
[READ], Return=OK, 300
{code}
And results with patch:
{code}
Run 1:
[OVERALL], RunTime(ms), 110798.0
[OVERALL], Throughput(ops/sec), 27076.30101626383
[READ], Operations, 300.0
[READ], AverageLatency(us), 2821.498163666
[READ], MinLatency(us), 186.0
[READ], MaxLatency(us), 266239.0
[READ], 95thPercentileLatency(us), 4423.0
[READ], 99thPercentileLatency(us), 6423.0
[READ], Return=OK, 300
Run 2:
[OVERALL], RunTime(ms), 101048.0
[OVERALL], Throughput(ops/sec), 29688.86073945056
[READ], Operations, 300.0
[READ], AverageLatency(us), 2471.022596332
[READ], MinLatency(us), 193.0
[READ], MaxLatency(us), 270335.0
[READ], 95thPercentileLatency(us), 4067.0
[READ], 99thPercentileLatency(us), 5123.0
[READ], Return=OK, 300
Run 3:
[OVERALL], RunTime(ms), 111897.0
[OVERALL], Throughput(ops/sec), 26810.370251213168
[READ], Operations, 300.0
[READ], AverageLatency(us), 2808.053357
[READ], MinLatency(us), 186.0
[READ], MaxLatency(us), 267775.0
[READ], 95thPercentileLatency(us), 4539.0
[READ], 99thPercentileLatency(us), 5947.0
[READ], Return=OK, 300
{code}
The avg runtime without the patch is {{107822.333}}, and with the patch
it is {{107914.333}}. From these results, there is no measured impact
doing ~28K ops/sec.
> Add per-user metrics
> -
>
> Key: HBASE-15519
> URL: https://issues.apache.org/jira/browse/HBASE-15519
> Project: HBase
> Issue Type: Sub-task
> Components: metrics
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: hbase-15519_v0.patch, hbase-15519_v1.patch
>
>
> Per-user metrics will be useful in multi-tenant cases where we can emit
> number of requests, operations, num RPCs etc at the per-user aggregate level
> per regionserver. We currently have throttles per user, but no way to monitor
> resource usage per-user.
> Looking at these metrics, operators can adjust throttles, do capacity
> planning, etc per-user.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15572) Adding optional timestamp semantics to HBase-Spark
[ https://issues.apache.org/jira/browse/HBASE-15572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated HBASE-15572: - Attachment: HBASE-15572-8.patch > Adding optional timestamp semantics to HBase-Spark > -- > > Key: HBASE-15572 > URL: https://issues.apache.org/jira/browse/HBASE-15572 > Project: HBase > Issue Type: New Feature > Components: spark >Reporter: Weiqing Yang >Assignee: Weiqing Yang > Fix For: 2.0.0 > > Attachments: HBASE-15572-1.patch, HBASE-15572-2.patch, > HBASE-15572-3.patch, HBASE-15572-4.patch, HBASE-15572-5.patch, > HBASE-15572-6.patch, HBASE-15572-7.patch, HBASE-15572-8.patch > > > Right now the timestamp is always latest. With this patch, users can select > timestamps they want. > In this patch, 4 parameters, "timestamp", "minTimestamp", "maxiTimestamp" and > "maxVersions" are added to HBaseSparkConf. Users can select a timestamp, they > can also select a time range with minimum timestamp and maximum timestamp. A > new test for selecting records with different timestamps is added. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15595) Document new metrics improvements
Enis Soztutar created HBASE-15595: - Summary: Document new metrics improvements Key: HBASE-15595 URL: https://issues.apache.org/jira/browse/HBASE-15595 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar We should document the improvements from the parent jira when we are done. Per-table, per-user metrics, dfs metrics, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15594) [YCSB] Improvements
[
https://issues.apache.org/jira/browse/HBASE-15594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226789#comment-15226789
]
Enis Soztutar commented on HBASE-15594:
---
Great. Coincidentally, I was running YCSB myself yesterday. I can try to
replicate your results.
> [YCSB] Improvements
> ---
>
> Key: HBASE-15594
> URL: https://issues.apache.org/jira/browse/HBASE-15594
> Project: HBase
> Issue Type: Umbrella
>Reporter: stack
>Priority: Critical
>
> Running YCSB and getting good results is an arcane art. For example, in my
> testing, a few handlers (100) with as many readers as I had CPUs (48), and
> upping connections on clients to same as #cpus made for 2-3x the throughput.
> The above config changes came of lore; which configurations need tweaking is
> not obvious going by their names, there were no indications from the app on
> where/why we were blocked or on which metrics are important to consider. Nor
> was any of this stuff written down in docs.
> Even still, I am stuck trying to make use of all of the machine. I am unable
> to overrun a server though 8 client nodes trying to beat up a single node
> (workloadc, all random-read, with no data returned -p readallfields=false).
> There is also a strange phenomenon where if I add a few machines, rather than
> 3x the YCSB throughput when 3 nodes in cluster, each machine instead is doing
> about 1/3rd.
> This umbrella issue is to host items that improve our defaults and noting how
> to get good numbers running YCSB. In particular, I want to be able to
> saturate a machine.
> Here are the configs I'm currently working with. I've not done the work to
> figure client-side if they are optimal (weird is how big a difference
> client-side changes can make -- need to fix this). On my 48 cpu machine, I
> can do about 370k random reads a second from data totally cached in
> bucketcache. If I short-circuit the user gets so they don't do any work but
> return immediately, I can do 600k ops a second but the CPUs are at 60-70%
> only. I cannot get them to go above this. Working on it.
> {code}
>
>
> hbase.ipc.server.read.threadpool.size
>
> 48
>
>
>
> hbase.regionserver.handler.count
>
> 100
>
>
>
> hbase.client.ipc.pool.size
>
> 100
>
>
>
> hbase.htable.threads.max
>
> 48
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15174) Client Public API should not have PB objects in 2.0
[
https://issues.apache.org/jira/browse/HBASE-15174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226778#comment-15226778
]
Enis Soztutar commented on HBASE-15174:
---
[~ram_krish] I was thinking that we can write a unit test for this so that it
does not happen again. The unit test can get the public method signatures of
all InterfaceAudience.Public classes and make sure that return values or
arguments are not PB. Something like TestInterfaceAudienceAnnotations. wdyt?
> Client Public API should not have PB objects in 2.0
> ---
>
> Key: HBASE-15174
> URL: https://issues.apache.org/jira/browse/HBASE-15174
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 2.0.0
>
>
> Some more cleanup for the parent jira.
> We have leaked some PB structs in Admin (and possible other places).
> We should clean up these API before 2.0.
> Examples include:
> {code}
> AdminProtos.GetRegionInfoResponse.CompactionState getCompactionState(final
> TableName tableName)
> throws IOException;
>
>
> void snapshot(final String snapshotName,
> final TableName tableName,
> HBaseProtos.SnapshotDescription.Type type) throws IOException,
> SnapshotCreationException,
> IllegalArgumentException;
>
> MasterProtos.SnapshotResponse
> takeSnapshotAsync(HBaseProtos.SnapshotDescription snapshot)
> throws IOException, SnapshotCreationException;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15572) Adding optional timestamp semantics to HBase-Spark
[
https://issues.apache.org/jira/browse/HBASE-15572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226747#comment-15226747
]
Hadoop QA commented on HBASE-15572:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 18s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 21s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 16s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 1m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 20s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 1m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 8s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 20s
{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 20s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12797118/HBASE-15572-7.patch |
| JIRA Issue | HBASE-15572 |
| Optional Tests | asflicense scalac scaladoc unit compile |
| uname | Linux asf910.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / a93a887 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/1294/testReport/ |
| modules | C: hbase-spark U: hbase-spark |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/1294/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> Adding optional timestamp semantics to HBase-Spark
> --
>
> Key: HBASE-15572
> URL: https://issues.apache.org/jira/browse/HBASE-15572
> Project: HBase
> Issue Type: New Feature
> Components: spark
>Reporter: Weiqing Yang
>Assignee: Weiqing Yang
> Fix For: 2.0.0
>
> Attachments: HBASE-15572-1.patch, HBASE-15572-2.patch,
> HBASE-15572-3.patch, HBASE-15572-4.patch, HBASE-15572-5.patch,
> HBASE-15572-6.patch, HBASE-15572-7.patch
>
>
> Right now the timestamp is always latest. With this patch, users can select
> timestamps they want.
> In this patch, 4 parameters, "timestamp", "minTimestamp", "maxiTimestamp" and
> "maxVersions" are added to HBaseSparkConf. Users can select a timestamp, they
> can also select a time range with minimum timestamp and maximum timestamp. A
> new test for selecting records with different timestamps is added.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-15174) Client Public API should not have PB objects in 2.0
[
https://issues.apache.org/jira/browse/HBASE-15174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan reassigned HBASE-15174:
--
Assignee: ramkrishna.s.vasudevan
> Client Public API should not have PB objects in 2.0
> ---
>
> Key: HBASE-15174
> URL: https://issues.apache.org/jira/browse/HBASE-15174
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 2.0.0
>
>
> Some more cleanup for the parent jira.
> We have leaked some PB structs in Admin (and possible other places).
> We should clean up these API before 2.0.
> Examples include:
> {code}
> AdminProtos.GetRegionInfoResponse.CompactionState getCompactionState(final
> TableName tableName)
> throws IOException;
>
>
> void snapshot(final String snapshotName,
> final TableName tableName,
> HBaseProtos.SnapshotDescription.Type type) throws IOException,
> SnapshotCreationException,
> IllegalArgumentException;
>
> MasterProtos.SnapshotResponse
> takeSnapshotAsync(HBaseProtos.SnapshotDescription snapshot)
> throws IOException, SnapshotCreationException;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14870) Backport namespace permissions to 98 branch
[ https://issues.apache.org/jira/browse/HBASE-14870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226696#comment-15226696 ] Hudson commented on HBASE-14870: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1197 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1197/]) HBASE-14870 Backport namespace permissions to 98 branch (apurtell: rev 379ec0debc8bc4e2a3f90a83304ddc0ec3cef060) * hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestNamespaceCommands.java * hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterAndRegionObserver.java * hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java * hbase-protocol/src/main/protobuf/Master.proto * hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterProtos.java > Backport namespace permissions to 98 branch > --- > > Key: HBASE-14870 > URL: https://issues.apache.org/jira/browse/HBASE-14870 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 0.98.19 > > Attachments: HBASE-14870-0.98.patch, HBASE-14870-0.98.patch > > > Backport namespace permissions to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15594) [YCSB] Improvements
[
https://issues.apache.org/jira/browse/HBASE-15594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226683#comment-15226683
]
Sean Busbey commented on HBASE-15594:
-
Is the plan for updated docs in our [very-outdated YCSB section in the ref
guide|http://hbase.apache.org/book.html#ycsb]? Or the [HBase README section in
the YCSB
project|https://github.com/brianfrankcooper/YCSB/blob/master/hbase098/README.md]?
either works since we can update the other to point. Since YCSB has multiple
clients to deal with multiple HBase API versions already, it might be easier to
point all of them at our ref guide.
> [YCSB] Improvements
> ---
>
> Key: HBASE-15594
> URL: https://issues.apache.org/jira/browse/HBASE-15594
> Project: HBase
> Issue Type: Umbrella
>Reporter: stack
>Priority: Critical
>
> Running YCSB and getting good results is an arcane art. For example, in my
> testing, a few handlers (100) with as many readers as I had CPUs (48), and
> upping connections on clients to same as #cpus made for 2-3x the throughput.
> The above config changes came of lore; which configurations need tweaking is
> not obvious going by their names, there were no indications from the app on
> where/why we were blocked or on which metrics are important to consider. Nor
> was any of this stuff written down in docs.
> Even still, I am stuck trying to make use of all of the machine. I am unable
> to overrun a server though 8 client nodes trying to beat up a single node
> (workloadc, all random-read, with no data returned -p readallfields=false).
> There is also a strange phenomenon where if I add a few machines, rather than
> 3x the YCSB throughput when 3 nodes in cluster, each machine instead is doing
> about 1/3rd.
> This umbrella issue is to host items that improve our defaults and noting how
> to get good numbers running YCSB. In particular, I want to be able to
> saturate a machine.
> Here are the configs I'm currently working with. I've not done the work to
> figure client-side if they are optimal (weird is how big a difference
> client-side changes can make -- need to fix this). On my 48 cpu machine, I
> can do about 370k random reads a second from data totally cached in
> bucketcache. If I short-circuit the user gets so they don't do any work but
> return immediately, I can do 600k ops a second but the CPUs are at 60-70%
> only. I cannot get them to go above this. Working on it.
> {code}
>
>
> hbase.ipc.server.read.threadpool.size
>
> 48
>
>
>
> hbase.regionserver.handler.count
>
> 100
>
>
>
> hbase.client.ipc.pool.size
>
> 100
>
>
>
> hbase.htable.threads.max
>
> 48
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15594) [YCSB] Improvements
[
https://issues.apache.org/jira/browse/HBASE-15594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226675#comment-15226675
]
stack commented on HBASE-15594:
---
I think HBASE-11425 helps us get more work done but the issue here is more
about getting over the apparent wall, improving defaults, and making configs
less cryptic.
> [YCSB] Improvements
> ---
>
> Key: HBASE-15594
> URL: https://issues.apache.org/jira/browse/HBASE-15594
> Project: HBase
> Issue Type: Umbrella
>Reporter: stack
>Priority: Critical
>
> Running YCSB and getting good results is an arcane art. For example, in my
> testing, a few handlers (100) with as many readers as I had CPUs (48), and
> upping connections on clients to same as #cpus made for 2-3x the throughput.
> The above config changes came of lore; which configurations need tweaking is
> not obvious going by their names, there were no indications from the app on
> where/why we were blocked or on which metrics are important to consider. Nor
> was any of this stuff written down in docs.
> Even still, I am stuck trying to make use of all of the machine. I am unable
> to overrun a server though 8 client nodes trying to beat up a single node
> (workloadc, all random-read, with no data returned -p readallfields=false).
> There is also a strange phenomenon where if I add a few machines, rather than
> 3x the YCSB throughput when 3 nodes in cluster, each machine instead is doing
> about 1/3rd.
> This umbrella issue is to host items that improve our defaults and noting how
> to get good numbers running YCSB. In particular, I want to be able to
> saturate a machine.
> Here are the configs I'm currently working with. I've not done the work to
> figure client-side if they are optimal (weird is how big a difference
> client-side changes can make -- need to fix this). On my 48 cpu machine, I
> can do about 370k random reads a second from data totally cached in
> bucketcache. If I short-circuit the user gets so they don't do any work but
> return immediately, I can do 600k ops a second but the CPUs are at 60-70%
> only. I cannot get them to go above this. Working on it.
> {code}
>
>
> hbase.ipc.server.read.threadpool.size
>
> 48
>
>
>
> hbase.regionserver.handler.count
>
> 100
>
>
>
> hbase.client.ipc.pool.size
>
> 100
>
>
>
> hbase.htable.threads.max
>
> 48
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15594) [YCSB] Improvements
[
https://issues.apache.org/jira/browse/HBASE-15594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226658#comment-15226658
]
Anoop Sam John commented on HBASE-15594:
Just wanted to know whether 11425 work helped wrt more CPU usage..We can
check this later also.
> [YCSB] Improvements
> ---
>
> Key: HBASE-15594
> URL: https://issues.apache.org/jira/browse/HBASE-15594
> Project: HBase
> Issue Type: Umbrella
>Reporter: stack
>Priority: Critical
>
> Running YCSB and getting good results is an arcane art. For example, in my
> testing, a few handlers (100) with as many readers as I had CPUs (48), and
> upping connections on clients to same as #cpus made for 2-3x the throughput.
> The above config changes came of lore; which configurations need tweaking is
> not obvious going by their names, there were no indications from the app on
> where/why we were blocked or on which metrics are important to consider. Nor
> was any of this stuff written down in docs.
> Even still, I am stuck trying to make use of all of the machine. I am unable
> to overrun a server though 8 client nodes trying to beat up a single node
> (workloadc, all random-read, with no data returned -p readallfields=false).
> There is also a strange phenomenon where if I add a few machines, rather than
> 3x the YCSB throughput when 3 nodes in cluster, each machine instead is doing
> about 1/3rd.
> This umbrella issue is to host items that improve our defaults and noting how
> to get good numbers running YCSB. In particular, I want to be able to
> saturate a machine.
> Here are the configs I'm currently working with. I've not done the work to
> figure client-side if they are optimal (weird is how big a difference
> client-side changes can make -- need to fix this). On my 48 cpu machine, I
> can do about 370k random reads a second from data totally cached in
> bucketcache. If I short-circuit the user gets so they don't do any work but
> return immediately, I can do 600k ops a second but the CPUs are at 60-70%
> only. I cannot get them to go above this. Working on it.
> {code}
>
>
> hbase.ipc.server.read.threadpool.size
>
> 48
>
>
>
> hbase.regionserver.handler.count
>
> 100
>
>
>
> hbase.client.ipc.pool.size
>
> 100
>
>
>
> hbase.htable.threads.max
>
> 48
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-15594) [YCSB] Improvements
[
https://issues.apache.org/jira/browse/HBASE-15594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226655#comment-15226655
]
stack edited comment on HBASE-15594 at 4/5/16 4:56 PM:
---
branch-1. branch-1 because I want to bring fixes into branch-1. You want me to
try something? [~anoop.hbase]
was (Author: stack):
branch-1. branch-1 because I want to bring fixes into branch-1. You want me to
try something?
> [YCSB] Improvements
> ---
>
> Key: HBASE-15594
> URL: https://issues.apache.org/jira/browse/HBASE-15594
> Project: HBase
> Issue Type: Umbrella
>Reporter: stack
>Priority: Critical
>
> Running YCSB and getting good results is an arcane art. For example, in my
> testing, a few handlers (100) with as many readers as I had CPUs (48), and
> upping connections on clients to same as #cpus made for 2-3x the throughput.
> The above config changes came of lore; which configurations need tweaking is
> not obvious going by their names, there were no indications from the app on
> where/why we were blocked or on which metrics are important to consider. Nor
> was any of this stuff written down in docs.
> Even still, I am stuck trying to make use of all of the machine. I am unable
> to overrun a server though 8 client nodes trying to beat up a single node
> (workloadc, all random-read, with no data returned -p readallfields=false).
> There is also a strange phenomenon where if I add a few machines, rather than
> 3x the YCSB throughput when 3 nodes in cluster, each machine instead is doing
> about 1/3rd.
> This umbrella issue is to host items that improve our defaults and noting how
> to get good numbers running YCSB. In particular, I want to be able to
> saturate a machine.
> Here are the configs I'm currently working with. I've not done the work to
> figure client-side if they are optimal (weird is how big a difference
> client-side changes can make -- need to fix this). On my 48 cpu machine, I
> can do about 370k random reads a second from data totally cached in
> bucketcache. If I short-circuit the user gets so they don't do any work but
> return immediately, I can do 600k ops a second but the CPUs are at 60-70%
> only. I cannot get them to go above this. Working on it.
> {code}
>
>
> hbase.ipc.server.read.threadpool.size
>
> 48
>
>
>
> hbase.regionserver.handler.count
>
> 100
>
>
>
> hbase.client.ipc.pool.size
>
> 100
>
>
>
> hbase.htable.threads.max
>
> 48
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15594) [YCSB] Improvements
[
https://issues.apache.org/jira/browse/HBASE-15594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226655#comment-15226655
]
stack commented on HBASE-15594:
---
branch-1. branch-1 because I want to bring fixes into branch-1. You want me to
try something?
> [YCSB] Improvements
> ---
>
> Key: HBASE-15594
> URL: https://issues.apache.org/jira/browse/HBASE-15594
> Project: HBase
> Issue Type: Umbrella
>Reporter: stack
>Priority: Critical
>
> Running YCSB and getting good results is an arcane art. For example, in my
> testing, a few handlers (100) with as many readers as I had CPUs (48), and
> upping connections on clients to same as #cpus made for 2-3x the throughput.
> The above config changes came of lore; which configurations need tweaking is
> not obvious going by their names, there were no indications from the app on
> where/why we were blocked or on which metrics are important to consider. Nor
> was any of this stuff written down in docs.
> Even still, I am stuck trying to make use of all of the machine. I am unable
> to overrun a server though 8 client nodes trying to beat up a single node
> (workloadc, all random-read, with no data returned -p readallfields=false).
> There is also a strange phenomenon where if I add a few machines, rather than
> 3x the YCSB throughput when 3 nodes in cluster, each machine instead is doing
> about 1/3rd.
> This umbrella issue is to host items that improve our defaults and noting how
> to get good numbers running YCSB. In particular, I want to be able to
> saturate a machine.
> Here are the configs I'm currently working with. I've not done the work to
> figure client-side if they are optimal (weird is how big a difference
> client-side changes can make -- need to fix this). On my 48 cpu machine, I
> can do about 370k random reads a second from data totally cached in
> bucketcache. If I short-circuit the user gets so they don't do any work but
> return immediately, I can do 600k ops a second but the CPUs are at 60-70%
> only. I cannot get them to go above this. Working on it.
> {code}
>
>
> hbase.ipc.server.read.threadpool.size
>
> 48
>
>
>
> hbase.regionserver.handler.count
>
> 100
>
>
>
> hbase.client.ipc.pool.size
>
> 100
>
>
>
> hbase.htable.threads.max
>
> 48
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15594) [YCSB] Improvements
[
https://issues.apache.org/jira/browse/HBASE-15594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226634#comment-15226634
]
Anoop Sam John commented on HBASE-15594:
So u are doing this test on trunk or branch-1?
> [YCSB] Improvements
> ---
>
> Key: HBASE-15594
> URL: https://issues.apache.org/jira/browse/HBASE-15594
> Project: HBase
> Issue Type: Umbrella
>Reporter: stack
>Priority: Critical
>
> Running YCSB and getting good results is an arcane art. For example, in my
> testing, a few handlers (100) with as many readers as I had CPUs (48), and
> upping connections on clients to same as #cpus made for 2-3x the throughput.
> The above config changes came of lore; which configurations need tweaking is
> not obvious going by their names, there were no indications from the app on
> where/why we were blocked or on which metrics are important to consider. Nor
> was any of this stuff written down in docs.
> Even still, I am stuck trying to make use of all of the machine. I am unable
> to overrun a server though 8 client nodes trying to beat up a single node
> (workloadc, all random-read, with no data returned -p readallfields=false).
> There is also a strange phenomenon where if I add a few machines, rather than
> 3x the YCSB throughput when 3 nodes in cluster, each machine instead is doing
> about 1/3rd.
> This umbrella issue is to host items that improve our defaults and noting how
> to get good numbers running YCSB. In particular, I want to be able to
> saturate a machine.
> Here are the configs I'm currently working with. I've not done the work to
> figure client-side if they are optimal (weird is how big a difference
> client-side changes can make -- need to fix this). On my 48 cpu machine, I
> can do about 370k random reads a second from data totally cached in
> bucketcache. If I short-circuit the user gets so they don't do any work but
> return immediately, I can do 600k ops a second but the CPUs are at 60-70%
> only. I cannot get them to go above this. Working on it.
> {code}
>
>
> hbase.ipc.server.read.threadpool.size
>
> 48
>
>
>
> hbase.regionserver.handler.count
>
> 100
>
>
>
> hbase.client.ipc.pool.size
>
> 100
>
>
>
> hbase.htable.threads.max
>
> 48
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15572) Adding optional timestamp semantics to HBase-Spark
[ https://issues.apache.org/jira/browse/HBASE-15572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated HBASE-15572: - Attachment: HBASE-15572-7.patch > Adding optional timestamp semantics to HBase-Spark > -- > > Key: HBASE-15572 > URL: https://issues.apache.org/jira/browse/HBASE-15572 > Project: HBase > Issue Type: New Feature > Components: spark >Reporter: Weiqing Yang >Assignee: Weiqing Yang > Fix For: 2.0.0 > > Attachments: HBASE-15572-1.patch, HBASE-15572-2.patch, > HBASE-15572-3.patch, HBASE-15572-4.patch, HBASE-15572-5.patch, > HBASE-15572-6.patch, HBASE-15572-7.patch > > > Right now the timestamp is always latest. With this patch, users can select > timestamps they want. > In this patch, 4 parameters, "timestamp", "minTimestamp", "maxiTimestamp" and > "maxVersions" are added to HBaseSparkConf. Users can select a timestamp, they > can also select a time range with minimum timestamp and maximum timestamp. A > new test for selecting records with different timestamps is added. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
[ https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15525: --- Fix Version/s: 2.0.0 > OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts > -- > > Key: HBASE-15525 > URL: https://issues.apache.org/jira/browse/HBASE-15525 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: deepankar >Assignee: Anoop Sam John >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-15525_V1.patch, HBASE-15525_WIP.patch, WIP.patch > > > After HBASE-13819 the system some times run out of direct memory whenever > there is some network congestion or some client side issues. > This was because of pending RPCs in the RPCServer$Connection.responseQueue > and since all the responses in this queue hold a buffer for cellblock from > BoundedByteBufferPool this could takeup a lot of memory if the > BoundedByteBufferPool's moving average settles down towards a higher value > See the discussion here > [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
[ https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15525: --- Status: Patch Available (was: Open) > OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts > -- > > Key: HBASE-15525 > URL: https://issues.apache.org/jira/browse/HBASE-15525 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: deepankar >Assignee: Anoop Sam John >Priority: Critical > Attachments: HBASE-15525_V1.patch, HBASE-15525_WIP.patch, WIP.patch > > > After HBASE-13819 the system some times run out of direct memory whenever > there is some network congestion or some client side issues. > This was because of pending RPCs in the RPCServer$Connection.responseQueue > and since all the responses in this queue hold a buffer for cellblock from > BoundedByteBufferPool this could takeup a lot of memory if the > BoundedByteBufferPool's moving average settles down towards a higher value > See the discussion here > [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
[ https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15525: --- Attachment: HBASE-15525_V1.patch [~dvdreddy] Can u pls help to test this in ur cluster > OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts > -- > > Key: HBASE-15525 > URL: https://issues.apache.org/jira/browse/HBASE-15525 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: deepankar >Assignee: Anoop Sam John >Priority: Critical > Attachments: HBASE-15525_V1.patch, HBASE-15525_WIP.patch, WIP.patch > > > After HBASE-13819 the system some times run out of direct memory whenever > there is some network congestion or some client side issues. > This was because of pending RPCs in the RPCServer$Connection.responseQueue > and since all the responses in this queue hold a buffer for cellblock from > BoundedByteBufferPool this could takeup a lot of memory if the > BoundedByteBufferPool's moving average settles down towards a higher value > See the discussion here > [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15594) [YCSB] Improvements
[
https://issues.apache.org/jira/browse/HBASE-15594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226612#comment-15226612
]
stack commented on HBASE-15594:
---
I think this a critical issue given we should be able to use all of a machine
and that YCSB is the yardstick by which we are measured (like-it-or-not)
> [YCSB] Improvements
> ---
>
> Key: HBASE-15594
> URL: https://issues.apache.org/jira/browse/HBASE-15594
> Project: HBase
> Issue Type: Umbrella
>Reporter: stack
>Priority: Critical
>
> Running YCSB and getting good results is an arcane art. For example, in my
> testing, a few handlers (100) with as many readers as I had CPUs (48), and
> upping connections on clients to same as #cpus made for 2-3x the throughput.
> The above config changes came of lore; which configurations need tweaking is
> not obvious going by their names, there were no indications from the app on
> where/why we were blocked or on which metrics are important to consider. Nor
> was any of this stuff written down in docs.
> Even still, I am stuck trying to make use of all of the machine. I am unable
> to overrun a server though 8 client nodes trying to beat up a single node
> (workloadc, all random-read, with no data returned -p readallfields=false).
> There is also a strange phenomenon where if I add a few machines, rather than
> 3x the YCSB throughput when 3 nodes in cluster, each machine instead is doing
> about 1/3rd.
> This umbrella issue is to host items that improve our defaults and noting how
> to get good numbers running YCSB. In particular, I want to be able to
> saturate a machine.
> Here are the configs I'm currently working with. I've not done the work to
> figure client-side if they are optimal (weird is how big a difference
> client-side changes can make -- need to fix this). On my 48 cpu machine, I
> can do about 370k random reads a second from data totally cached in
> bucketcache. If I short-circuit the user gets so they don't do any work but
> return immediately, I can do 600k ops a second but the CPUs are at 60-70%
> only. I cannot get them to go above this. Working on it.
> {code}
>
>
> hbase.ipc.server.read.threadpool.size
>
> 48
>
>
>
> hbase.regionserver.handler.count
>
> 100
>
>
>
> hbase.client.ipc.pool.size
>
> 100
>
>
>
> hbase.htable.threads.max
>
> 48
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14855) Connect to regionserver
[ https://issues.apache.org/jira/browse/HBASE-14855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14855: -- Attachment: HBASE-14855-v3.patch Put the sample client in it's own binary and better control over linking stuff. > Connect to regionserver > --- > > Key: HBASE-14855 > URL: https://issues.apache.org/jira/browse/HBASE-14855 > Project: HBase > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-14855-v2.patch, HBASE-14855-v3.patch, > HBASE-14855.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15594) [YCSB] Improvements
[
https://issues.apache.org/jira/browse/HBASE-15594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-15594:
--
Priority: Critical (was: Major)
> [YCSB] Improvements
> ---
>
> Key: HBASE-15594
> URL: https://issues.apache.org/jira/browse/HBASE-15594
> Project: HBase
> Issue Type: Umbrella
>Reporter: stack
>Priority: Critical
>
> Running YCSB and getting good results is an arcane art. For example, in my
> testing, a few handlers (100) with as many readers as I had CPUs (48), and
> upping connections on clients to same as #cpus made for 2-3x the throughput.
> The above config changes came of lore; which configurations need tweaking is
> not obvious going by their names, there were no indications from the app on
> where/why we were blocked or on which metrics are important to consider. Nor
> was any of this stuff written down in docs.
> Even still, I am stuck trying to make use of all of the machine. I am unable
> to overrun a server though 8 client nodes trying to beat up a single node
> (workloadc, all random-read, with no data returned -p readallfields=false).
> There is also a strange phenomenon where if I add a few machines, rather than
> 3x the YCSB throughput when 3 nodes in cluster, each machine instead is doing
> about 1/3rd.
> This umbrella issue is to host items that improve our defaults and noting how
> to get good numbers running YCSB. In particular, I want to be able to
> saturate a machine.
> Here are the configs I'm currently working with. I've not done the work to
> figure client-side if they are optimal (weird is how big a difference
> client-side changes can make -- need to fix this). On my 48 cpu machine, I
> can do about 370k random reads a second from data totally cached in
> bucketcache. If I short-circuit the user gets so they don't do any work but
> return immediately, I can do 600k ops a second but the CPUs are at 60-70%
> only. I cannot get them to go above this. Working on it.
> {code}
>
>
> hbase.ipc.server.read.threadpool.size
>
> 48
>
>
>
> hbase.regionserver.handler.count
>
> 100
>
>
>
> hbase.client.ipc.pool.size
>
> 100
>
>
>
> hbase.htable.threads.max
>
> 48
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-15594) [YCSB] Improvements
stack created HBASE-15594:
-
Summary: [YCSB] Improvements
Key: HBASE-15594
URL: https://issues.apache.org/jira/browse/HBASE-15594
Project: HBase
Issue Type: Umbrella
Reporter: stack
Running YCSB and getting good results is an arcane art. For example, in my
testing, a few handlers (100) with as many readers as I had CPUs (48), and
upping connections on clients to same as #cpus made for 2-3x the throughput.
The above config changes came of lore; which configurations need tweaking is
not obvious going by their names, there were no indications from the app on
where/why we were blocked or on which metrics are important to consider. Nor
was any of this stuff written down in docs.
Even still, I am stuck trying to make use of all of the machine. I am unable to
overrun a server though 8 client nodes trying to beat up a single node
(workloadc, all random-read, with no data returned -p readallfields=false).
There is also a strange phenomenon where if I add a few machines, rather than
3x the YCSB throughput when 3 nodes in cluster, each machine instead is doing
about 1/3rd.
This umbrella issue is to host items that improve our defaults and noting how
to get good numbers running YCSB. In particular, I want to be able to saturate
a machine.
Here are the configs I'm currently working with. I've not done the work to
figure client-side if they are optimal (weird is how big a difference
client-side changes can make -- need to fix this). On my 48 cpu machine, I can
do about 370k random reads a second from data totally cached in bucketcache. If
I short-circuit the user gets so they don't do any work but return immediately,
I can do 600k ops a second but the CPUs are at 60-70% only. I cannot get them
to go above this. Working on it.
{code}
hbase.ipc.server.read.threadpool.size
48
hbase.regionserver.handler.count
100
hbase.client.ipc.pool.size
100
hbase.htable.threads.max
48
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15400) Use DateTieredCompactor for Date Tiered Compaction
[ https://issues.apache.org/jira/browse/HBASE-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Clara Xiong updated HBASE-15400: Attachment: HBASE-15400-v3-v5.patch More updated for HadoopQA. Test failure is unrelated and passes locally. > Use DateTieredCompactor for Date Tiered Compaction > -- > > Key: HBASE-15400 > URL: https://issues.apache.org/jira/browse/HBASE-15400 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Clara Xiong >Assignee: Clara Xiong > Fix For: 2.0.0 > > Attachments: HBASE-15400-15389-v12.patch, HBASE-15400-v1.pa, > HBASE-15400-v3-v3.patch, HBASE-15400-v3-v4.patch, HBASE-15400-v3-v5.patch, > HBASE-15400-v3.patch, HBASE-15400.patch > > > When we compact, we can output multiple files along the current window > boundaries. There are two use cases: > 1. Major compaction: We want to output date tiered store files with data > older than max age archived in trunks of the window size on the higher tier. > Once a window is old enough, we don't combine the windows to promote to the > next tier any further. So files in these windows retain the same timespan as > they were minor-compacted last time, which is the window size of the highest > tier. Major compaction will touch these files and we want to maintain the > same layout. This way, TTL and archiving will be simpler and more efficient. > 2. Bulk load files and the old file generated by major compaction before > upgrading to DTCP. > Pros: > 1. Restore locality, process versioning, updates and deletes while > maintaining the tiered layout. > 2. The best way to fix a skewed layout. > > This work is based on a prototype of DateTieredCompactor from HBASE-15389 and > focused on the part to meet needs for these two use cases while supporting > others. I have to call out a few design decisions: > 1. We only want to output the files along all windows for major compaction. > And we want to output multiple files older than max age in the trunks of the > maximum tier window size determined by base window size, windows per tier and > max age. > 2. For minor compaction, we don't want to output too many files, which will > remain around because of current restriction of contiguous compaction by seq > id. I will only output two files if all the files in the windows are being > combined, one for the data within window and the other for the out-of-window > tail. If there is any file in the window excluded from compaction, only one > file will be output from compaction. When the windows are promoted, the > situation of out of order data will gradually improve. For the incoming > window, we need to accommodate the case with user-specified future data. > 3. We have to pass the boundaries with the list of store file as a complete > time snapshot instead of two separate calls because window layout is > determined by the time the computation is called. So we will need new type of > compaction request. > 4. Since we will assign the same seq id for all output files, we need to sort > by maxTimestamp subsequently. Right now all compaction policy gets the files > sorted for StoreFileManager which sorts by seq id and other criteria. I will > use this order for DTCP only, to avoid impacting other compaction policies. > 5. We need some cleanup of current design of StoreEngine and CompactionPolicy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-15481) Add pre/post roll to WALObserver
[ https://issues.apache.org/jira/browse/HBASE-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi reassigned HBASE-15481: --- Assignee: (was: Matteo Bertozzi) > Add pre/post roll to WALObserver > > > Key: HBASE-15481 > URL: https://issues.apache.org/jira/browse/HBASE-15481 > Project: HBase > Issue Type: New Feature >Affects Versions: 1.3.0 >Reporter: Matteo Bertozzi >Priority: Trivial > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15481-v0.patch, HBASE-15481-v1.patch, > HBASE-15481-v1.patch, HBASE-15481-v2.patch > > > currently the WALObserver has only a pre/post Write. It will be useful to > have a pre/post Roll too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15537) Add multi WAL support for AsyncFSWAL
[
https://issues.apache.org/jira/browse/HBASE-15537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226562#comment-15226562
]
stack commented on HBASE-15537:
---
bq. Is it enough to get the statistics you want?
Yes. I can try it. This is just generally helpful I'd say.
Also, yeah, agree that we should first prove we have an overlap problem; lets
see how it runs. Let me try a test. Doesn't have to be before commit.
> Add multi WAL support for AsyncFSWAL
>
>
> Key: HBASE-15537
> URL: https://issues.apache.org/jira/browse/HBASE-15537
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15537-v3.patch, HBASE-15537-v4.patch,
> HBASE-15537-v5.patch, HBASE-15537.patch, HBASE-15537_v2.patch
>
>
> The multi WAL should not be bound with {{FSHLog}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15572) Adding optional timestamp semantics to HBase-Spark
[ https://issues.apache.org/jira/browse/HBASE-15572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226546#comment-15226546 ] Ted Yu commented on HBASE-15572: [~busbey]: Mind taking a look at the latest patch ? > Adding optional timestamp semantics to HBase-Spark > -- > > Key: HBASE-15572 > URL: https://issues.apache.org/jira/browse/HBASE-15572 > Project: HBase > Issue Type: New Feature > Components: spark >Reporter: Weiqing Yang >Assignee: Weiqing Yang > Fix For: 2.0.0 > > Attachments: HBASE-15572-1.patch, HBASE-15572-2.patch, > HBASE-15572-3.patch, HBASE-15572-4.patch, HBASE-15572-5.patch, > HBASE-15572-6.patch > > > Right now the timestamp is always latest. With this patch, users can select > timestamps they want. > In this patch, 4 parameters, "timestamp", "minTimestamp", "maxiTimestamp" and > "maxVersions" are added to HBaseSparkConf. Users can select a timestamp, they > can also select a time range with minimum timestamp and maximum timestamp. A > new test for selecting records with different timestamps is added. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15572) Adding optional timestamp semantics to HBase-Spark
[ https://issues.apache.org/jira/browse/HBASE-15572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15572: --- Release Note: Right now the timestamp is always latest. With this patch, users can select timestamps they want. In this patch, 4 parameters, "timestamp", "minTimestamp", "maxiTimestamp" and "maxVersions" are added to HBaseSparkConf. Users can select a timestamp, they can also select a time range with minimum timestamp and maximum timestamp. was: Right now the timestamp is always latest. With this patch, users can select timestamps they want. In this patch, 4 parameters, "timestamp", "minTimestamp", "maxiTimestamp" and "maxVersions" are added to HBaseSparkConf. Users can select a timestamp, they can also select a time range with minimum timestamp and maximum timestamp. A new test for selecting records with different timestamps is added. > Adding optional timestamp semantics to HBase-Spark > -- > > Key: HBASE-15572 > URL: https://issues.apache.org/jira/browse/HBASE-15572 > Project: HBase > Issue Type: New Feature > Components: spark >Reporter: Weiqing Yang >Assignee: Weiqing Yang > Fix For: 2.0.0 > > Attachments: HBASE-15572-1.patch, HBASE-15572-2.patch, > HBASE-15572-3.patch, HBASE-15572-4.patch, HBASE-15572-5.patch, > HBASE-15572-6.patch > > > Right now the timestamp is always latest. With this patch, users can select > timestamps they want. > In this patch, 4 parameters, "timestamp", "minTimestamp", "maxiTimestamp" and > "maxVersions" are added to HBaseSparkConf. Users can select a timestamp, they > can also select a time range with minimum timestamp and maximum timestamp. A > new test for selecting records with different timestamps is added. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15537) Add multi WAL support for AsyncFSWAL
[
https://issues.apache.org/jira/browse/HBASE-15537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226209#comment-15226209
]
Sean Busbey commented on HBASE-15537:
-
I don't think we need any changes on the HBase side to get the stats that
[~stack] wants. When doing the Cloudera-internal testing for multiwal and HDFS
HSM, I gathered these stats by running HDFS fsck in a loop to get block
locations for all files under the WAL directory. It worked fine, from what I
remember.
Unfortunately, I didn't save the logs from that work.
> Add multi WAL support for AsyncFSWAL
>
>
> Key: HBASE-15537
> URL: https://issues.apache.org/jira/browse/HBASE-15537
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15537-v3.patch, HBASE-15537-v4.patch,
> HBASE-15537-v5.patch, HBASE-15537.patch, HBASE-15537_v2.patch
>
>
> The multi WAL should not be bound with {{FSHLog}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14870) Backport namespace permissions to 98 branch
[ https://issues.apache.org/jira/browse/HBASE-14870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226189#comment-15226189 ] Hudson commented on HBASE-14870: FAILURE: Integrated in HBase-0.98-matrix #324 (See [https://builds.apache.org/job/HBase-0.98-matrix/324/]) HBASE-14870 Backport namespace permissions to 98 branch (apurtell: rev 379ec0debc8bc4e2a3f90a83304ddc0ec3cef060) * hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java * hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestNamespaceCommands.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * hbase-protocol/src/main/protobuf/Master.proto * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterProtos.java * hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterAndRegionObserver.java * hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java * hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java > Backport namespace permissions to 98 branch > --- > > Key: HBASE-14870 > URL: https://issues.apache.org/jira/browse/HBASE-14870 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 0.98.19 > > Attachments: HBASE-14870-0.98.patch, HBASE-14870-0.98.patch > > > Backport namespace permissions to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15593) Time limit of scanning should be offered by client
[ https://issues.apache.org/jira/browse/HBASE-15593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226185#comment-15226185 ] Phil Yang commented on HBASE-15593: --- This optimization will be very useful when RS is slow and many requests are waiting. > Time limit of scanning should be offered by client > -- > > Key: HBASE-15593 > URL: https://issues.apache.org/jira/browse/HBASE-15593 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Phil Yang >Assignee: Phil Yang > > In RSRpcServices.scan, we will set a time limit equaling to > Math.min(scannerLeaseTimeoutPeriod, rpcTimeout) / 2, and will response > heartbeat message if we reach this limit. However, two timeout settings > (hbase.client.scanner.timeout.period and hbase.rpc.timeout) are read from > RS's configure, which may be different from client's. If client's setting is > much less than server's, there may still be timeout at client side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15593) Time limit of scanning should be offered by client
[ https://issues.apache.org/jira/browse/HBASE-15593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-15593: -- Description: In RSRpcServices.scan, we will set a time limit equaling to Math.min(scannerLeaseTimeoutPeriod, rpcTimeout) / 2, and will response heartbeat message if we reach this limit. However, two timeout settings (hbase.client.scanner.timeout.period and hbase.rpc.timeout) are read from RS's configure, which may be different from client's. If client's setting is much less than server's, there may still be timeout at client side. (was: In RSRpcServices.scan, we will set a time limit equaling to Math.min(scannerLeaseTimeoutPeriod, rpcTimeout) / 2, and will response heartbeat message if we reach this limit. However, two timeout settings (hbase.client.scanner.timeout.period and hbase.rpc.timeout) is read from RS's configure, which may be different from client's. If client's setting is much less than server's, there may still be timeout at client side.) > Time limit of scanning should be offered by client > -- > > Key: HBASE-15593 > URL: https://issues.apache.org/jira/browse/HBASE-15593 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Phil Yang >Assignee: Phil Yang > > In RSRpcServices.scan, we will set a time limit equaling to > Math.min(scannerLeaseTimeoutPeriod, rpcTimeout) / 2, and will response > heartbeat message if we reach this limit. However, two timeout settings > (hbase.client.scanner.timeout.period and hbase.rpc.timeout) are read from > RS's configure, which may be different from client's. If client's setting is > much less than server's, there may still be timeout at client side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15593) Time limit of scanning should be offered by client
[ https://issues.apache.org/jira/browse/HBASE-15593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226176#comment-15226176 ] Phil Yang commented on HBASE-15593: --- And a more general question: Will HBase RPC request contain a timeout from client? I don't find it but I'm not familiar with HBase RPC. If so we can use it; if not, there will be an optimization that we can drop rpc requests which are already timeout. > Time limit of scanning should be offered by client > -- > > Key: HBASE-15593 > URL: https://issues.apache.org/jira/browse/HBASE-15593 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Phil Yang >Assignee: Phil Yang > > In RSRpcServices.scan, we will set a time limit equaling to > Math.min(scannerLeaseTimeoutPeriod, rpcTimeout) / 2, and will response > heartbeat message if we reach this limit. However, two timeout settings > (hbase.client.scanner.timeout.period and hbase.rpc.timeout) is read from RS's > configure, which may be different from client's. If client's setting is much > less than server's, there may still be timeout at client side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15593) Time limit of scanning should be offered by client
Phil Yang created HBASE-15593: - Summary: Time limit of scanning should be offered by client Key: HBASE-15593 URL: https://issues.apache.org/jira/browse/HBASE-15593 Project: HBase Issue Type: Bug Affects Versions: 1.1.0 Reporter: Phil Yang Assignee: Phil Yang In RSRpcServices.scan, we will set a time limit equaling to Math.min(scannerLeaseTimeoutPeriod, rpcTimeout) / 2, and will response heartbeat message if we reach this limit. However, two timeout settings (hbase.client.scanner.timeout.period and hbase.rpc.timeout) is read from RS's configure, which may be different from client's. If client's setting is much less than server's, there may still be timeout at client side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15537) Add multi WAL support for AsyncFSWAL
[
https://issues.apache.org/jira/browse/HBASE-15537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226139#comment-15226139
]
Yu Li commented on HBASE-15537:
---
bq. If we have an existing WAL pipeline, can we pass the DN participants as
excluded when NN goes to make up the next pipeline?
We could pass in some favored nodes when creating output stream, but
unfortunately there's no API for excluding nodes... maybe we could give it a
try to see how the auto load-balance works before trying to get full control?
> Add multi WAL support for AsyncFSWAL
>
>
> Key: HBASE-15537
> URL: https://issues.apache.org/jira/browse/HBASE-15537
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15537-v3.patch, HBASE-15537-v4.patch,
> HBASE-15537-v5.patch, HBASE-15537.patch, HBASE-15537_v2.patch
>
>
> The multi WAL should not be bound with {{FSHLog}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15400) Use DateTieredCompactor for Date Tiered Compaction
[
https://issues.apache.org/jira/browse/HBASE-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226033#comment-15226033
]
Hadoop QA commented on HBASE-15400:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 45s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 8m
0s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 4s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 51s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 13m 8s
{color} | {color:red} hbase-server: patch generated 2 new + 110 unchanged - 28
fixed = 112 total (was 138) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
18s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
134m 28s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 176m 7s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
2s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 361m 13s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.balancer.TestStochasticLoadBalancer2
|
| Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestMobSecureExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestExportSnapshot |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12796981/HBASE-15400-v3-v4.patch
|
| JIRA Issue | HBASE-15400 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64
[jira] [Commented] (HBASE-15485) Filter.reset() should not be called between batches
[ https://issues.apache.org/jira/browse/HBASE-15485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225949#comment-15225949 ] Hudson commented on HBASE-15485: FAILURE: Integrated in HBase-1.1-JDK7 #1695 (See [https://builds.apache.org/job/HBase-1.1-JDK7/1695/]) HBASE-15485 Filter.reset() should not be called between batches (Phil (tedyu: rev 734f196b32a4b10c915f74357ffd894cbbb2b430) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScannerContext.java * hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilterFromRegionSide.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > Filter.reset() should not be called between batches > --- > > Key: HBASE-15485 > URL: https://issues.apache.org/jira/browse/HBASE-15485 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.1.3 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.5, 1.2.2 > > Attachments: HBASE-15485-branch-1-v2.patch, > HBASE-15485-branch-1-v3.patch, HBASE-15485-branch-1.1-v1.patch, > HBASE-15485-branch-1.1-v2.patch, HBASE-15485-branch-1.patch, > HBASE-15485-v1.patch, HBASE-15485-v2.patch, HBASE-15485-v3.patch, > HBASE-15485-v4.patch, HBASE-15485-v5.patch > > > As discussed in HBASE-15325, now we will resetFilters if partial result not > formed, but we should not reset filters when batch limit reached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15537) Add multi WAL support for AsyncFSWAL
[
https://issues.apache.org/jira/browse/HBASE-15537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225934#comment-15225934
]
Hadoop QA commented on HBASE-15537:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 5m 46s
{color} | {color:red} hbase-server: patch generated 2 new + 11 unchanged - 1
fixed = 13 total (was 12) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 28s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 128m 33s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 189m 54s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.security.access.TestNamespaceCommands |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12796987/HBASE-15537-v5.patch |
| JIRA Issue | HBASE-15537 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux asf910.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / a93a887 |
| Default Java | 1.7.0_79 |
| Multi-JDK versions | /home/jenkins/tools/java/jdk1.8.0:1.8.0
[jira] [Commented] (HBASE-15485) Filter.reset() should not be called between batches
[ https://issues.apache.org/jira/browse/HBASE-15485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225818#comment-15225818 ] Hudson commented on HBASE-15485: SUCCESS: Integrated in HBase-1.1-JDK8 #1781 (See [https://builds.apache.org/job/HBase-1.1-JDK8/1781/]) HBASE-15485 Filter.reset() should not be called between batches (Phil (tedyu: rev 734f196b32a4b10c915f74357ffd894cbbb2b430) * hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilterFromRegionSide.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScannerContext.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > Filter.reset() should not be called between batches > --- > > Key: HBASE-15485 > URL: https://issues.apache.org/jira/browse/HBASE-15485 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.1.3 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.5, 1.2.2 > > Attachments: HBASE-15485-branch-1-v2.patch, > HBASE-15485-branch-1-v3.patch, HBASE-15485-branch-1.1-v1.patch, > HBASE-15485-branch-1.1-v2.patch, HBASE-15485-branch-1.patch, > HBASE-15485-v1.patch, HBASE-15485-v2.patch, HBASE-15485-v3.patch, > HBASE-15485-v4.patch, HBASE-15485-v5.patch > > > As discussed in HBASE-15325, now we will resetFilters if partial result not > formed, but we should not reset filters when batch limit reached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15485) Filter.reset() should not be called between batches
[ https://issues.apache.org/jira/browse/HBASE-15485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225814#comment-15225814 ] Hudson commented on HBASE-15485: FAILURE: Integrated in HBase-1.4 #70 (See [https://builds.apache.org/job/HBase-1.4/70/]) HBASE-15485 Filter.reset() should not be called between batches (Phil (tedyu: rev 8957dc1033c2731594e397d3e532be0231b2a1ba) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilterFromRegionSide.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScannerContext.java > Filter.reset() should not be called between batches > --- > > Key: HBASE-15485 > URL: https://issues.apache.org/jira/browse/HBASE-15485 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.1.3 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.5, 1.2.2 > > Attachments: HBASE-15485-branch-1-v2.patch, > HBASE-15485-branch-1-v3.patch, HBASE-15485-branch-1.1-v1.patch, > HBASE-15485-branch-1.1-v2.patch, HBASE-15485-branch-1.patch, > HBASE-15485-v1.patch, HBASE-15485-v2.patch, HBASE-15485-v3.patch, > HBASE-15485-v4.patch, HBASE-15485-v5.patch > > > As discussed in HBASE-15325, now we will resetFilters if partial result not > formed, but we should not reset filters when batch limit reached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15485) Filter.reset() should not be called between batches
[ https://issues.apache.org/jira/browse/HBASE-15485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225797#comment-15225797 ] Hudson commented on HBASE-15485: SUCCESS: Integrated in HBase-1.2 #597 (See [https://builds.apache.org/job/HBase-1.2/597/]) HBASE-15485 Filter.reset() should not be called between batches (Phil (tedyu: rev 70447aa5f4860e4d48e68b3e9a880ad58d57b937) * hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilterFromRegionSide.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScannerContext.java > Filter.reset() should not be called between batches > --- > > Key: HBASE-15485 > URL: https://issues.apache.org/jira/browse/HBASE-15485 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.1.3 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.5, 1.2.2 > > Attachments: HBASE-15485-branch-1-v2.patch, > HBASE-15485-branch-1-v3.patch, HBASE-15485-branch-1.1-v1.patch, > HBASE-15485-branch-1.1-v2.patch, HBASE-15485-branch-1.patch, > HBASE-15485-v1.patch, HBASE-15485-v2.patch, HBASE-15485-v3.patch, > HBASE-15485-v4.patch, HBASE-15485-v5.patch > > > As discussed in HBASE-15325, now we will resetFilters if partial result not > formed, but we should not reset filters when batch limit reached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15485) Filter.reset() should not be called between batches
[ https://issues.apache.org/jira/browse/HBASE-15485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225783#comment-15225783 ] Hudson commented on HBASE-15485: SUCCESS: Integrated in HBase-1.3-IT #596 (See [https://builds.apache.org/job/HBase-1.3-IT/596/]) HBASE-15485 Filter.reset() should not be called between batches (Phil (tedyu: rev 8957dc1033c2731594e397d3e532be0231b2a1ba) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScannerContext.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilterFromRegionSide.java > Filter.reset() should not be called between batches > --- > > Key: HBASE-15485 > URL: https://issues.apache.org/jira/browse/HBASE-15485 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.1.3 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.5, 1.2.2 > > Attachments: HBASE-15485-branch-1-v2.patch, > HBASE-15485-branch-1-v3.patch, HBASE-15485-branch-1.1-v1.patch, > HBASE-15485-branch-1.1-v2.patch, HBASE-15485-branch-1.patch, > HBASE-15485-v1.patch, HBASE-15485-v2.patch, HBASE-15485-v3.patch, > HBASE-15485-v4.patch, HBASE-15485-v5.patch > > > As discussed in HBASE-15325, now we will resetFilters if partial result not > formed, but we should not reset filters when batch limit reached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15485) Filter.reset() should not be called between batches
[ https://issues.apache.org/jira/browse/HBASE-15485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225758#comment-15225758 ] Hudson commented on HBASE-15485: FAILURE: Integrated in HBase-1.3 #634 (See [https://builds.apache.org/job/HBase-1.3/634/]) HBASE-15485 Filter.reset() should not be called between batches (Phil (tedyu: rev ba88a42efad0c4a46bad0be57a6c5ae86961da2b) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilterFromRegionSide.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScannerContext.java > Filter.reset() should not be called between batches > --- > > Key: HBASE-15485 > URL: https://issues.apache.org/jira/browse/HBASE-15485 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.1.3 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.5, 1.2.2 > > Attachments: HBASE-15485-branch-1-v2.patch, > HBASE-15485-branch-1-v3.patch, HBASE-15485-branch-1.1-v1.patch, > HBASE-15485-branch-1.1-v2.patch, HBASE-15485-branch-1.patch, > HBASE-15485-v1.patch, HBASE-15485-v2.patch, HBASE-15485-v3.patch, > HBASE-15485-v4.patch, HBASE-15485-v5.patch > > > As discussed in HBASE-15325, now we will resetFilters if partial result not > formed, but we should not reset filters when batch limit reached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12663) unify getTableDescriptors() and listTableDescriptorsByNamespace()
[ https://issues.apache.org/jira/browse/HBASE-12663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-12663: --- Fix Version/s: 0.98.19 > unify getTableDescriptors() and listTableDescriptorsByNamespace() > - > > Key: HBASE-12663 > URL: https://issues.apache.org/jira/browse/HBASE-12663 > Project: HBase > Issue Type: Bug > Components: master, security >Affects Versions: 1.0.0, 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 1.0.0, 2.0.0, 0.98.19 > > Attachments: HBASE-12663-v0.patch > > > currently we have getTableDescriptors()/listTableNames() > and listTableDescriptorsByNamespace()/listTableNamesByNamespace() > which are just filtering by namespace. > The first two are already able to filter tables by regex, > and there is also an ACL check which shows only the table the user have > access to, > while the namespace version just return a list without any ACLs check. > to me there is no point to keep the *ByNamespace() version on the server > side, > since the "base" version must be able to do the filtering by > namespace and the ACLs check must be the same for both anyway. > or at least we can just call the base version with the ns filtering. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14809) Grant / revoke Namespace admin permission to group
[
https://issues.apache.org/jira/browse/HBASE-14809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-14809:
---
Fix Version/s: 0.98.19
> Grant / revoke Namespace admin permission to group
> ---
>
> Key: HBASE-14809
> URL: https://issues.apache.org/jira/browse/HBASE-14809
> Project: HBase
> Issue Type: Bug
> Components: security
>Affects Versions: 1.0.2
>Reporter: Steven Hancz
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3, 0.98.19
>
> Attachments: 14809-v1.txt, 14809-v2.txt, 14809-v3.txt, 14809-v3.txt,
> 14809-v4.txt
>
>
> Hi,
> We are looking to roll out HBase and are in the process to design the
> security model.
> We are looking to implement global DBAs and Namespace specific
> administrators.
> So for example the global dba would create a namespace and grant a user/group
> admin privileges within that ns.
> So that a given ns admin can in turn create objects and grant permission
> within the given ns only.
> We have run into some issues at the ns admin level. It appears that a ns
> admin can NOT grant to a grop unless it also has global admin privilege. But
> once it has global admin privilege it can grant in any NS not just the one
> where it has admin privileges.
> Based on the HBase documentation at
> http://hbase.apache.org/book.html#appendix_acl_matrix
> Table 13. ACL Matrix
> Interface Operation Permissions
> AccessController grant(global level) global(A)
> grant(namespace level) global(A)|NS(A)
> grant at a namespace level should be possible for someone with global A OR
> (|) NS A permission.
> As you will see in our test it does not work if NS A permission is granted
> but global A permission is not.
> Here you can see that group hbaseappltest_ns1admin has XCA permission on ns1.
> {code}
> hbase(main):011:0> scan 'hbase:acl'
> ROW COLUMN+CELL
> @ns1 column=l:@hbaseappltest_ns1admin, timestamp=1446676679787, value=XCA
> {code}
> However:
> Here you can see that a user who is member of the group
> hbaseappltest_ns1admin can not grant a WRX privilege to a group as it is
> missing global A privilege.
> {code}
> $hbase shell
> 15/11/13 10:02:23 INFO Configuration.deprecation: hadoop.native.lib is
> deprecated. Instead, use io.native.lib.available
> HBase Shell; enter 'help' for list of supported commands.
> Type "exit" to leave the HBase Shell
> Version 1.0.0-cdh5.4.7, rUnknown, Thu Sep 17 02:25:03 PDT 2015
> hbase(main):001:0> whoami
> ns1ad...@wlab.net (auth:KERBEROS)
> groups: hbaseappltest_ns1admin
> hbase(main):002:0> grant '@hbaseappltest_ns1funct' ,'RWX','@ns1'
> ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'ns1admin' (global, action=ADMIN)
> {code}
> The way I read the documentation a NS admin should be able to grant as it has
> ns level A privilege not only object level permission.
> CDH is a version 5.4.7 and Hbase is version 1.0.
> Regards,
> Steven
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12674) Add permission check to getNamespaceDescriptor()
[ https://issues.apache.org/jira/browse/HBASE-12674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-12674: --- Fix Version/s: 0.98.19 > Add permission check to getNamespaceDescriptor() > > > Key: HBASE-12674 > URL: https://issues.apache.org/jira/browse/HBASE-12674 > Project: HBase > Issue Type: Bug > Components: master, security >Affects Versions: 1.0.0, 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 1.0.0, 2.0.0, 0.98.19 > > Attachments: HBASE-12674-v0.patch > > > Add permission check to getNamespaceDescriptor() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14870) Backport namespace permissions to 98 branch
[ https://issues.apache.org/jira/browse/HBASE-14870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-14870: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Backport namespace permissions to 98 branch > --- > > Key: HBASE-14870 > URL: https://issues.apache.org/jira/browse/HBASE-14870 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 0.98.19 > > Attachments: HBASE-14870-0.98.patch, HBASE-14870-0.98.patch > > > Backport namespace permissions to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12511) namespace permissions - add support from table creation privilege in a namespace 'C'
[ https://issues.apache.org/jira/browse/HBASE-12511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-12511: --- Fix Version/s: 0.98.19 > namespace permissions - add support from table creation privilege in a > namespace 'C' > > > Key: HBASE-12511 > URL: https://issues.apache.org/jira/browse/HBASE-12511 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Huaiyu Zhu >Priority: Blocker > Fix For: 1.0.0, 2.0.0, 1.1.0, 0.98.19 > > Attachments: HBASE-12511-addendum.patch, HBASE-12511.patch, > hbase-12511_v5.patch, hbase-12511_v6.patch, hbase-12511_v7.patch, > hbase-12511_v8.patch, hbase-12511_v9.patch > > > As discussed in namespace permission Jira. A user granted a 'C' on a > namespace enables a user to create tables within the namespace. 'C' on a > namespace does not enable a user to create/drop the namespace. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
