[jira] [Commented] (HBASE-15519) Add per-user metrics
[ https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15207043#comment-15207043 ] Elliott Clark commented on HBASE-15519: --- Anything that has to do a map lookup for metrics is awful. We should make sure that we never add that back. We got a 20% improvement by removing map lookups in the metrics path. > Add per-user metrics > - > > Key: HBASE-15519 > URL: https://issues.apache.org/jira/browse/HBASE-15519 > Project: HBase > Issue Type: Sub-task > Components: metrics >Reporter: Enis Soztutar > > Per-user metrics will be useful in multi-tenant cases where we can emit > number of requests, operations, num RPCs etc at the per-user aggregate level > per regionserver. We currently have throttles per user, but no way to monitor > resource usage per-user. > Looking at these metrics, operators can adjust throttles, do capacity > planning, etc per-user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[ https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15207259#comment-15207259 ] Enis Soztutar commented on HBASE-15519: --- bq. We should make sure that we never add that back. We got a 20% improvement by removing map lookups in the metrics path. Agreed that we should be careful in the perf analysis to make sure that hot code paths are not affected. Some overhead was due to long string concatenations for coming up with the metric name I think. We also check the quotas per-operation, so something to check. > Add per-user metrics > - > > Key: HBASE-15519 > URL: https://issues.apache.org/jira/browse/HBASE-15519 > Project: HBase > Issue Type: Sub-task > Components: metrics >Reporter: Enis Soztutar > > Per-user metrics will be useful in multi-tenant cases where we can emit > number of requests, operations, num RPCs etc at the per-user aggregate level > per regionserver. We currently have throttles per user, but no way to monitor > resource usage per-user. > Looking at these metrics, operators can adjust throttles, do capacity > planning, etc per-user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[
https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15231912#comment-15231912
]
Hadoop QA commented on HBASE-15519:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 33s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 44s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
6s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 6s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 9s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 48s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 23s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 44s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 10s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 45s
{color} | {color:red} hbase-hadoop2-compat generated 1 new + 0 unchanged - 0
fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 45s
{color} | {color:red} hbase-hadoop2-compat generated 1 new + 0 unchanged - 0
fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 15s
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 15s
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s
{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s
{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {
[jira] [Commented] (HBASE-15519) Add per-user metrics
[
https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15233314#comment-15233314
]
Hadoop QA commented on HBASE-15519:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 35s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 9s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 36s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
6s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 11m
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 18s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 15s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 15s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 55s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
51m 5s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 13m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 55s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 10s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 26s
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 40s
{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 44s
{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 124m 50s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 50s {color}
| {color:red} hbase-ser
[jira] [Commented] (HBASE-15519) Add per-user metrics
[
https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16958451#comment-16958451
]
Hudson commented on HBASE-15519:
Results for branch branch-2
[build #2332 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2332/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2332//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2332//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2332//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add per-user metrics
> -
>
> Key: HBASE-15519
> URL: https://issues.apache.org/jira/browse/HBASE-15519
> Project: HBase
> Issue Type: Sub-task
> Components: metrics
>Affects Versions: 1.2.0
>Reporter: Enis Soztutar
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch,
> hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch
>
>
> Per-user metrics will be useful in multi-tenant cases where we can emit
> number of requests, operations, num RPCs etc at the per-user aggregate level
> per regionserver. We currently have throttles per user, but no way to monitor
> resource usage per-user.
> Looking at these metrics, operators can adjust throttles, do capacity
> planning, etc per-user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[
https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16958959#comment-16958959
]
Hudson commented on HBASE-15519:
Results for branch master
[build #1515 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1515/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1515//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1515//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1515//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add per-user metrics
> -
>
> Key: HBASE-15519
> URL: https://issues.apache.org/jira/browse/HBASE-15519
> Project: HBase
> Issue Type: Sub-task
> Components: metrics
>Affects Versions: 1.2.0
>Reporter: Enis Soztutar
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch,
> hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch
>
>
> Per-user metrics will be useful in multi-tenant cases where we can emit
> number of requests, operations, num RPCs etc at the per-user aggregate level
> per regionserver. We currently have throttles per user, but no way to monitor
> resource usage per-user.
> Looking at these metrics, operators can adjust throttles, do capacity
> planning, etc per-user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[
https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961559#comment-16961559
]
Hudson commented on HBASE-15519:
Results for branch branch-2
[build #2337 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2337/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2337//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2337//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2337//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add per-user metrics
> -
>
> Key: HBASE-15519
> URL: https://issues.apache.org/jira/browse/HBASE-15519
> Project: HBase
> Issue Type: Sub-task
> Components: metrics
>Affects Versions: 1.2.0
>Reporter: Enis Soztutar
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch,
> hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch
>
>
> Per-user metrics will be useful in multi-tenant cases where we can emit
> number of requests, operations, num RPCs etc at the per-user aggregate level
> per regionserver. We currently have throttles per user, but no way to monitor
> resource usage per-user.
> Looking at these metrics, operators can adjust throttles, do capacity
> planning, etc per-user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[
https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961886#comment-16961886
]
Hudson commented on HBASE-15519:
Results for branch branch-1
[build #1121 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1121/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1121//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1121//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/1121//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Add per-user metrics
> -
>
> Key: HBASE-15519
> URL: https://issues.apache.org/jira/browse/HBASE-15519
> Project: HBase
> Issue Type: Sub-task
> Components: metrics
>Affects Versions: 1.2.0
>Reporter: Enis Soztutar
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch,
> hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch
>
>
> Per-user metrics will be useful in multi-tenant cases where we can emit
> number of requests, operations, num RPCs etc at the per-user aggregate level
> per regionserver. We currently have throttles per user, but no way to monitor
> resource usage per-user.
> Looking at these metrics, operators can adjust throttles, do capacity
> planning, etc per-user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[
https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962092#comment-16962092
]
Hudson commented on HBASE-15519:
Results for branch master
[build #1520 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1520/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1520//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1520//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1520//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add per-user metrics
> -
>
> Key: HBASE-15519
> URL: https://issues.apache.org/jira/browse/HBASE-15519
> Project: HBase
> Issue Type: Sub-task
> Components: metrics
>Affects Versions: 1.2.0
>Reporter: Enis Soztutar
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch,
> hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch
>
>
> Per-user metrics will be useful in multi-tenant cases where we can emit
> number of requests, operations, num RPCs etc at the per-user aggregate level
> per regionserver. We currently have throttles per user, but no way to monitor
> resource usage per-user.
> Looking at these metrics, operators can adjust throttles, do capacity
> planning, etc per-user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[ https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16915841#comment-16915841 ] Sean Busbey commented on HBASE-15519: - this topic came up again in reviews of HBASE-11062. I think we need testing with more users to see what the load on a RS looks like. test with > 16 users, > 100 users, > 1000 users. test with long tail users (like 4 users who are around a lot and then a few hundred who only show up with freq < metric system refresh) an implementation should be based on the lossy counting of top-X that got added in HBASE-19722 / HBASE-21991. > Add per-user metrics > - > > Key: HBASE-15519 > URL: https://issues.apache.org/jira/browse/HBASE-15519 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 1.2.0 >Reporter: Enis Soztutar >Assignee: Enis Soztutar >Priority: Major > Attachments: hbase-15519_v0.patch, hbase-15519_v1.patch, > hbase-15519_v1.patch, hbase-15519_v2.patch > > > Per-user metrics will be useful in multi-tenant cases where we can emit > number of requests, operations, num RPCs etc at the per-user aggregate level > per regionserver. We currently have throttles per user, but no way to monitor > resource usage per-user. > Looking at these metrics, operators can adjust throttles, do capacity > planning, etc per-user. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[
https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933857#comment-16933857
]
Ankit Singhal commented on HBASE-15519:
---
[~busbey], attaching forward port of the v2 patch and adding lossy counting for
the user metrics. 003.patch is on top of HBASE-23054.
{quote}I think we need testing with more users to see what the load on a RS
looks like. test with > 16 users, > 100 users, > 1000 users. test with long
tail users (like 4 users who are around a lot and then a few hundred who only
show up with freq < metric system refresh)
{quote}
Let me know if we still need scale testing when lossyCounting is implemented.
> Add per-user metrics
> -
>
> Key: HBASE-15519
> URL: https://issues.apache.org/jira/browse/HBASE-15519
> Project: HBase
> Issue Type: Sub-task
> Components: metrics
>Affects Versions: 1.2.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Major
> Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch,
> hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch
>
>
> Per-user metrics will be useful in multi-tenant cases where we can emit
> number of requests, operations, num RPCs etc at the per-user aggregate level
> per regionserver. We currently have throttles per user, but no way to monitor
> resource usage per-user.
> Looking at these metrics, operators can adjust throttles, do capacity
> planning, etc per-user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[ https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934257#comment-16934257 ] Andrew Purtell commented on HBASE-15519: Pardon the potentially ignorant question. I'm on vacation with just a phone. Does this work include exporting at least a subset of per user accouting in ClusterStatus? It would be fine to do this as a follow up of course. Reason I ask is hbtop receives its telemetry from ClusterStatus and an obvious and important enhancement for that tool would be a display mode for top N user or top N client activity by server or region (or table or namespace aggregates). > Add per-user metrics > - > > Key: HBASE-15519 > URL: https://issues.apache.org/jira/browse/HBASE-15519 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 1.2.0 >Reporter: Enis Soztutar >Assignee: Enis Soztutar >Priority: Major > Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch, > hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch > > > Per-user metrics will be useful in multi-tenant cases where we can emit > number of requests, operations, num RPCs etc at the per-user aggregate level > per regionserver. We currently have throttles per user, but no way to monitor > resource usage per-user. > Looking at these metrics, operators can adjust throttles, do capacity > planning, etc per-user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[
https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16936303#comment-16936303
]
Ankit Singhal commented on HBASE-15519:
---
{quote}Pardon the potentially ignorant question. I'm on vacation with just a
phone. Does this work include exporting at least a subset of per user accouting
in ClusterStatus? It would be fine to do this as a follow up of course. Reason
I ask is hbtop receives its telemetry from ClusterStatus and an obvious and
important enhancement for that tool would be a display mode for top N user or
top N client activity by server or region (or table or namespace aggregates).
{quote}
sorry for the late response, currently these metrics are not added in
clusterMetrics/ServerMetrics but we can add them. I see you created HBASE-23065
, so let me take up that and work on it.
And also LossyCounting keeps metrics for those users only which are active(and
new) and trim others, which help in saving some network while transferring them
and make more sense for topN user or topN client use-case as well (though have
importance at a point in time only).
> Add per-user metrics
> -
>
> Key: HBASE-15519
> URL: https://issues.apache.org/jira/browse/HBASE-15519
> Project: HBase
> Issue Type: Sub-task
> Components: metrics
>Affects Versions: 1.2.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Major
> Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch,
> hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch
>
>
> Per-user metrics will be useful in multi-tenant cases where we can emit
> number of requests, operations, num RPCs etc at the per-user aggregate level
> per regionserver. We currently have throttles per user, but no way to monitor
> resource usage per-user.
> Looking at these metrics, operators can adjust throttles, do capacity
> planning, etc per-user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[ https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16936391#comment-16936391 ] Andrew Purtell commented on HBASE-15519: Point in time is suitable for the hbtop case and even for display in shell “status ‘detailed’” output, so that lines up. Thank you. Agree also the other issue is good for tracking the needed ClusterStatus changes. > Add per-user metrics > - > > Key: HBASE-15519 > URL: https://issues.apache.org/jira/browse/HBASE-15519 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 1.2.0 >Reporter: Enis Soztutar >Assignee: Enis Soztutar >Priority: Major > Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch, > hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch > > > Per-user metrics will be useful in multi-tenant cases where we can emit > number of requests, operations, num RPCs etc at the per-user aggregate level > per regionserver. We currently have throttles per user, but no way to monitor > resource usage per-user. > Looking at these metrics, operators can adjust throttles, do capacity > planning, etc per-user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[ https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16937197#comment-16937197 ] Sean Busbey commented on HBASE-15519: - Thanks for working on this! Can you post your change as a PR? It's gotten big enough that the patch file is awkward. > Add per-user metrics > - > > Key: HBASE-15519 > URL: https://issues.apache.org/jira/browse/HBASE-15519 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 1.2.0 >Reporter: Enis Soztutar >Assignee: Enis Soztutar >Priority: Major > Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch, > hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch > > > Per-user metrics will be useful in multi-tenant cases where we can emit > number of requests, operations, num RPCs etc at the per-user aggregate level > per regionserver. We currently have throttles per user, but no way to monitor > resource usage per-user. > Looking at these metrics, operators can adjust throttles, do capacity > planning, etc per-user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[ https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938010#comment-16938010 ] Ankit Singhal commented on HBASE-15519: --- Thanks [~busbey] for offering help with the review. Have created a pull request #661 for the same. Thanks [~apurtell] as well for the feedback. > Add per-user metrics > - > > Key: HBASE-15519 > URL: https://issues.apache.org/jira/browse/HBASE-15519 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 1.2.0 >Reporter: Enis Soztutar >Assignee: Enis Soztutar >Priority: Major > Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch, > hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch > > > Per-user metrics will be useful in multi-tenant cases where we can emit > number of requests, operations, num RPCs etc at the per-user aggregate level > per regionserver. We currently have throttles per user, but no way to monitor > resource usage per-user. > Looking at these metrics, operators can adjust throttles, do capacity > planning, etc per-user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[ https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17175882#comment-17175882 ] Nick Dimiduk commented on HBASE-15519: -- Was this reopened by mistake? The feature has been released, subsequent changes will require subsequent Jira issues. Thanks. > Add per-user metrics > - > > Key: HBASE-15519 > URL: https://issues.apache.org/jira/browse/HBASE-15519 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 1.2.0 >Reporter: Enis Soztutar >Assignee: Ankit Singhal >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch, > hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch > > > Per-user metrics will be useful in multi-tenant cases where we can emit > number of requests, operations, num RPCs etc at the per-user aggregate level > per regionserver. We currently have throttles per user, but no way to monitor > resource usage per-user. > Looking at these metrics, operators can adjust throttles, do capacity > planning, etc per-user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[ https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163919#comment-17163919 ] Michael Stack commented on HBASE-15519: --- [~an...@apache.org] There are two users accessing my system but I have 138 instances of a thread whose name is always 'lossy-count-0". This seems like a mistake? [~xucang] ... flagging you because the basis here is your work (though off by default). Thanks. > Add per-user metrics > - > > Key: HBASE-15519 > URL: https://issues.apache.org/jira/browse/HBASE-15519 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 1.2.0 >Reporter: Enis Soztutar >Assignee: Ankit Singhal >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch, > hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch > > > Per-user metrics will be useful in multi-tenant cases where we can emit > number of requests, operations, num RPCs etc at the per-user aggregate level > per regionserver. We currently have throttles per user, but no way to monitor > resource usage per-user. > Looking at these metrics, operators can adjust throttles, do capacity > planning, etc per-user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[ https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163943#comment-17163943 ] Michael Stack commented on HBASE-15519: --- As I read this, we have per-user metrics enabled per Region. Each instance creates an executor which has a single thread to do lossy-counting. This seems profligate? No? It shouldn't be default, do ye think? > Add per-user metrics > - > > Key: HBASE-15519 > URL: https://issues.apache.org/jira/browse/HBASE-15519 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 1.2.0 >Reporter: Enis Soztutar >Assignee: Ankit Singhal >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch, > hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch > > > Per-user metrics will be useful in multi-tenant cases where we can emit > number of requests, operations, num RPCs etc at the per-user aggregate level > per regionserver. We currently have throttles per user, but no way to monitor > resource usage per-user. > Looking at these metrics, operators can adjust throttles, do capacity > planning, etc per-user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[ https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163953#comment-17163953 ] Michael Stack commented on HBASE-15519: --- Does this feature work? I see the unit tests but on a real cluster where I have many users, in these metrics, the user shows as 'nobody' or 'hbase'... > Add per-user metrics > - > > Key: HBASE-15519 > URL: https://issues.apache.org/jira/browse/HBASE-15519 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 1.2.0 >Reporter: Enis Soztutar >Assignee: Ankit Singhal >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch, > hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch > > > Per-user metrics will be useful in multi-tenant cases where we can emit > number of requests, operations, num RPCs etc at the per-user aggregate level > per regionserver. We currently have throttles per user, but no way to monitor > resource usage per-user. > Looking at these metrics, operators can adjust throttles, do capacity > planning, etc per-user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[
https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163961#comment-17163961
]
Ankit Singhal commented on HBASE-15519:
---
{quote}Each instance creates an executor which has a single thread to do
lossy-counting. This seems profligate? No?
{quote}
Yes Looks like a miss by me, we could share an executor with more threads and a
non-blocking queue here.
{quote}Does this feature work? I see the unit tests but on a real cluster where
I have many users, in these metrics, the user shows as 'nobody' or 'hbase'...
{quote}
I tested it with few users on my local setup and I was able to see the users
appearing with the right request count when the requests are coming and
disappears as there is no request, Idea was to get only the top users at a
time. Let me recollect my knowledge around lossyCounting and see how the users
are removed.
> Add per-user metrics
> -
>
> Key: HBASE-15519
> URL: https://issues.apache.org/jira/browse/HBASE-15519
> Project: HBase
> Issue Type: Sub-task
> Components: metrics
>Affects Versions: 1.2.0
>Reporter: Enis Soztutar
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
> Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch,
> hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch
>
>
> Per-user metrics will be useful in multi-tenant cases where we can emit
> number of requests, operations, num RPCs etc at the per-user aggregate level
> per regionserver. We currently have throttles per user, but no way to monitor
> resource usage per-user.
> Looking at these metrics, operators can adjust throttles, do capacity
> planning, etc per-user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[ https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163978#comment-17163978 ] Michael Stack commented on HBASE-15519: --- Thanks for the prompt response [~an...@apache.org] The lossy-count thread spin up seems extreme at least. How we figure user name of incoming client seems like it could get some improvement: i.e. instead of 'hbase', perhaps 'hbase.ip' or 'hbase.hostname'... Metrics have bloated now I look. This is not main offender (though it could grow to be if lots of users?) but I see between 35-45 MBeans and then between 2M and 3M for the list of all metrics... which seems like a lot. This looks like a really useful feature but on by default might be a bit much. Ideal I think would be being able to switch it on when a problem but then off again given its expense But it doesn't seem possible given the way our metrics work it seems. Thanks for entertaining my questions. > Add per-user metrics > - > > Key: HBASE-15519 > URL: https://issues.apache.org/jira/browse/HBASE-15519 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 1.2.0 >Reporter: Enis Soztutar >Assignee: Ankit Singhal >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch, > hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch > > > Per-user metrics will be useful in multi-tenant cases where we can emit > number of requests, operations, num RPCs etc at the per-user aggregate level > per regionserver. We currently have throttles per user, but no way to monitor > resource usage per-user. > Looking at these metrics, operators can adjust throttles, do capacity > planning, etc per-user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[ https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163980#comment-17163980 ] Andrew Kyle Purtell commented on HBASE-15519: - Can we make lossy counting housekeeping global to the LossyCounting class (or a helper class) so the overhead is constant no matter how many instances? We already accept imprecision so having one thread service them all should be reasonable... or maybe some small thread pool sized logarithmically with respect to the total number of instances? > Add per-user metrics > - > > Key: HBASE-15519 > URL: https://issues.apache.org/jira/browse/HBASE-15519 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 1.2.0 >Reporter: Enis Soztutar >Assignee: Ankit Singhal >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch, > hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch > > > Per-user metrics will be useful in multi-tenant cases where we can emit > number of requests, operations, num RPCs etc at the per-user aggregate level > per regionserver. We currently have throttles per user, but no way to monitor > resource usage per-user. > Looking at these metrics, operators can adjust throttles, do capacity > planning, etc per-user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[ https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17164035#comment-17164035 ] Michael Stack commented on HBASE-15519: --- Let me set the default as off. It seems like the lossy counters can come and go. In a little test cluster I see they rise and fall but even still.. I see instances of 40 threads which seems a bunch for counting. In my prod situation, there's a steady 140+ threads doing lossycounting which is a bit over the top. > Add per-user metrics > - > > Key: HBASE-15519 > URL: https://issues.apache.org/jira/browse/HBASE-15519 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 1.2.0 >Reporter: Enis Soztutar >Assignee: Ankit Singhal >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch, > hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch > > > Per-user metrics will be useful in multi-tenant cases where we can emit > number of requests, operations, num RPCs etc at the per-user aggregate level > per regionserver. We currently have throttles per user, but no way to monitor > resource usage per-user. > Looking at these metrics, operators can adjust throttles, do capacity > planning, etc per-user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[
https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17164471#comment-17164471
]
Josh Elser commented on HBASE-15519:
{quote}I see instances of 40 threads which seems a bunch for counting. In my
prod situation, there's a steady 140+ threads doing lossycounting which is a
bit over the top.
{quote}
Yeah, that seems over-kill. Off by default is fine to resolve quickly, IMO
Your data being shared here is great, Stack (2-3M metrics, 35-45MBeans). As
much data you can give to make a representative "at-scale" would help in making
we don't fall into the same trap would be great. Ideally, we come back here and
clean this up and make it OK for on-by-default.
> Add per-user metrics
> -
>
> Key: HBASE-15519
> URL: https://issues.apache.org/jira/browse/HBASE-15519
> Project: HBase
> Issue Type: Sub-task
> Components: metrics
>Affects Versions: 1.2.0
>Reporter: Enis Soztutar
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
> Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch,
> hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch
>
>
> Per-user metrics will be useful in multi-tenant cases where we can emit
> number of requests, operations, num RPCs etc at the per-user aggregate level
> per regionserver. We currently have throttles per user, but no way to monitor
> resource usage per-user.
> Looking at these metrics, operators can adjust throttles, do capacity
> planning, etc per-user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[ https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17164476#comment-17164476 ] Michael Stack commented on HBASE-15519: --- [~elserj] I meant 2-3 MB (to be clear) – ~300 Regions per host on the cluster I'm looking at (we are doing metrics per Region by default). Thanks Josh. > Add per-user metrics > - > > Key: HBASE-15519 > URL: https://issues.apache.org/jira/browse/HBASE-15519 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 1.2.0 >Reporter: Enis Soztutar >Assignee: Ankit Singhal >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch, > hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch > > > Per-user metrics will be useful in multi-tenant cases where we can emit > number of requests, operations, num RPCs etc at the per-user aggregate level > per regionserver. We currently have throttles per user, but no way to monitor > resource usage per-user. > Looking at these metrics, operators can adjust throttles, do capacity > planning, etc per-user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[
https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17164477#comment-17164477
]
Josh Elser commented on HBASE-15519:
hahaha, ok. Thank you for clarifying. I went a bit bug-eyed at that number of
metrics :)
{quote}How we figure user name of incoming client seems like it could get some
improvement: i.e. instead of 'hbase', perhaps 'hbase.ip' or 'hbase.hostname'...
{quote}
I'm curious what this looks like for you, too? One/few client IP's, or many
from a bunch of hosts?
> Add per-user metrics
> -
>
> Key: HBASE-15519
> URL: https://issues.apache.org/jira/browse/HBASE-15519
> Project: HBase
> Issue Type: Sub-task
> Components: metrics
>Affects Versions: 1.2.0
>Reporter: Enis Soztutar
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
> Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch,
> hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch
>
>
> Per-user metrics will be useful in multi-tenant cases where we can emit
> number of requests, operations, num RPCs etc at the per-user aggregate level
> per regionserver. We currently have throttles per user, but no way to monitor
> resource usage per-user.
> Looking at these metrics, operators can adjust throttles, do capacity
> planning, etc per-user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[ https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17164496#comment-17164496 ] Michael Stack commented on HBASE-15519: --- For me, on one cluster I see 'hbase' and the name of the application; two users total though in both cases the origins are multiple... this is how they identify. On another cluster with lots of different applications running, there is one user only – 'nobody'. > Add per-user metrics > - > > Key: HBASE-15519 > URL: https://issues.apache.org/jira/browse/HBASE-15519 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 1.2.0 >Reporter: Enis Soztutar >Assignee: Ankit Singhal >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch, > hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch > > > Per-user metrics will be useful in multi-tenant cases where we can emit > number of requests, operations, num RPCs etc at the per-user aggregate level > per regionserver. We currently have throttles per user, but no way to monitor > resource usage per-user. > Looking at these metrics, operators can adjust throttles, do capacity > planning, etc per-user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[
https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17165945#comment-17165945
]
Michael Stack commented on HBASE-15519:
---
So, yeah, the per-user I think needs working on. For example, here is what I
see for connections... i.e. all from one 'user', the APP_USER, all lumped
together, though from different IPs:
{code:java}
2020-07-27 17:21:49,576 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.189.210:33190, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:49,615 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.184.233:37032, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:49,992 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.188.92:42026, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:50,136 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.178.154:40568, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:50,894 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.185.40:59874, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:51,082 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.188.86:46592, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:51,470 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.181.25:56220, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:51,478 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.184.211:44888, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:51,644 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.185.34:45848, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:51,954 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.174.87:46344, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:52,342 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.191.23:40736, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:52,461 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.181.150:39626, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:52,494 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.186.83:44636, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:52,663 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.181.211:40950, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService
2020-07-27 17:21:52,996 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.192.184.147:37758, version=1.2.0, sasl=false, ugi=APP_NAME
(auth:SIMPLE), service=ClientService {code}
> Add per-user metrics
> -
>
> Key: HBASE-15519
> URL: https://issues.apache.org/jira/browse/HBASE-15519
> Project: HBase
> Issue Type: Sub-task
> Components: metrics
>Affects Versions: 1.2.0
>Reporter: Enis Soztutar
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
> Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch,
> hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch
>
>
> Per-user metrics will be useful in multi-tenant cases where we can emit
> number of requests, operations, num RPCs etc at the per-user aggregate level
> per regionserver. We currently have throttles per user, but no way to monitor
> resource usage per-user.
> Looking at these metrics, operators can adjust throttles, do capacity
> planning, etc per-user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[
https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17170508#comment-17170508
]
Hudson commented on HBASE-15519:
Results for branch branch-2.3
[build #198 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/198/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/198/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/198/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/198/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/198/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add per-user metrics
> -
>
> Key: HBASE-15519
> URL: https://issues.apache.org/jira/browse/HBASE-15519
> Project: HBase
> Issue Type: Sub-task
> Components: metrics
>Affects Versions: 1.2.0
>Reporter: Enis Soztutar
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
> Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch,
> hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch
>
>
> Per-user metrics will be useful in multi-tenant cases where we can emit
> number of requests, operations, num RPCs etc at the per-user aggregate level
> per regionserver. We currently have throttles per user, but no way to monitor
> resource usage per-user.
> Looking at these metrics, operators can adjust throttles, do capacity
> planning, etc per-user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[
https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17170550#comment-17170550
]
Hudson commented on HBASE-15519:
Results for branch branch-2
[build #2767 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2767/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2767/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2767//console].
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2758/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2758/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add per-user metrics
> -
>
> Key: HBASE-15519
> URL: https://issues.apache.org/jira/browse/HBASE-15519
> Project: HBase
> Issue Type: Sub-task
> Components: metrics
>Affects Versions: 1.2.0
>Reporter: Enis Soztutar
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
> Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch,
> hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch
>
>
> Per-user metrics will be useful in multi-tenant cases where we can emit
> number of requests, operations, num RPCs etc at the per-user aggregate level
> per regionserver. We currently have throttles per user, but no way to monitor
> resource usage per-user.
> Looking at these metrics, operators can adjust throttles, do capacity
> planning, etc per-user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[
https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17171522#comment-17171522
]
Hudson commented on HBASE-15519:
Results for branch master
[build #1804 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1804/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1804/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1804/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1804/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/master/1804//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Add per-user metrics
> -
>
> Key: HBASE-15519
> URL: https://issues.apache.org/jira/browse/HBASE-15519
> Project: HBase
> Issue Type: Sub-task
> Components: metrics
>Affects Versions: 1.2.0
>Reporter: Enis Soztutar
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
> Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch,
> hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch
>
>
> Per-user metrics will be useful in multi-tenant cases where we can emit
> number of requests, operations, num RPCs etc at the per-user aggregate level
> per regionserver. We currently have throttles per user, but no way to monitor
> resource usage per-user.
> Looking at these metrics, operators can adjust throttles, do capacity
> planning, etc per-user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
