[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497670#comment-16497670
]
liubangchen commented on HBASE-20654:
-
I unstand it . Thanks
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: 1.png, HBASE-20654-1.patch, HBASE-20654-2.patch,
> HBASE-20654-3.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497572#comment-16497572
]
stack commented on HBASE-20654:
---
bq. . We hope quickly find these failded open or failed close and show at web
console in order to assign these regions again by users .
The Master home page shows RITs already? See
./org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon. See how it will
list RIT if any. See the
./org/apache/hadoop/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon page.
If you want to show RIT elsewhere, use the shell as per Sean above or ask the
Master from remote webconsole. Do
admin.getClusterMetrics(ClusterMetrics.Option.REGIONS_IN_TRANSITION).. It
passes you back a ClusterMetrics instance on which you can do
#getRegionStatesInTransition and it will return List where
RegionState has region names, status, etc. (If need it in hbase1, use old
ClusterStatus object)?
Metrics in hbase are used to keep counts; not data such as region names.
Keeping arbitrary Strings that come and go will bloat our metrics output and
will always be truncated. Metrics are generated on a period regardless of
whether they are read or not whereas asking hbase when you need to draw a
webconsole page is a one-time affair.
Does this help?
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: 1.png, HBASE-20654-1.patch, HBASE-20654-2.patch,
> HBASE-20654-3.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497531#comment-16497531
]
liubangchen commented on HBASE-20654:
-
When hbase as a PAAS service,users can not login any server.But when
regionserver restart or crash some region may me in state failed open or failed
close ,then client will get exception NotServingRegionException . But it very
difficult to find these failed open regions with out use hbck . We hope quickly
find these failded open or failed close and show at web console in order to
assign these regions again by users .
We gather metrics by GangliaSink31 or jmx , and our case as follow:
!1.png!
logs on server
{code:java}
2017-07-08 08:43:54,821 WARN [AM.-pool1-t138] master.RegionStates: Failed to
open/close 3ecca5a149b8813ce97cd97629775f07 on
emr-10-112-99-101,16010,1497344643126, set to FAILED_CLOSE
2017-07-08 08:43:54,821 WARN [AM.-pool1-t130] master.RegionStates: Failed to
open/close 4535a4fb157ccff13e5da275cc79a952 on
emr-10-112-99-101,16010,1497344643126, set to FAILED_CLOSE
{code}
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: 1.png, HBASE-20654-1.patch, HBASE-20654-2.patch,
> HBASE-20654-3.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497410#comment-16497410
]
Josh Elser commented on HBASE-20654:
{quote}Please say more than this. Counts of RITs makes sense as a metric and to
expose as JMX or even stuff like longest RIT.
{quote}
+1 I agree with Stack here. At most, we could include a list of tables which
have RITs in JMX. I think we should be trying to provide information to alert
admins that there is a problem through JMX not trying to provide information
that would let someone fully debug a cluster problem (e.g. user reports to
admin "my jobs are timing out", admin checks JMX "Oh, the table your job
queries has an RIT", admin digs to take action and user waits patiently)
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch, HBASE-20654-2.patch,
> HBASE-20654-3.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496859#comment-16496859
]
stack commented on HBASE-20654:
---
bq. If expose these regions state info to jxm is very useful.
Please say more than this. Counts of RITs makes sense as a metric and to expose
as JMX or even stuff like longest RIT. I am having trouble understanding why
we need explicit names exposed in metrics. If a problem I'd imagine an operator
will notice it via review of RIT metric counts and when they go to deal with a
specific RIT, they'll figure exact region by reading WAL/log or shell output as
per Sean above or just ask hbck for list of RIT names.
Correct me if I'm wrong. I do not see why our metrics need to be fattened up
with long lists of region names. If a cluster of 1M regions on 1K nodes, at
startup, generating such a metrics few will be crippling.
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch, HBASE-20654-2.patch,
> HBASE-20654-3.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496156#comment-16496156
]
Hadoop QA commented on HBASE-20654:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
48s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
48s{color} | {color:blue} hbase-server in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} hbase-hadoop2-compat: The patch generated 1 new + 4
unchanged - 1 fixed = 5 total (was 5) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
7s{color} | {color:red} hbase-server: The patch generated 5 new + 38 unchanged
- 0 fixed = 43 total (was 38) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
55s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 58s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
6s{color} | {color:red} hbase-server generated 1 new + 2 unchanged - 0 fixed =
3 total (was 2) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
26s{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 28s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 66m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Inconsistent synchronization of
org.apache.hadoop.hbase.master.assignment.MetricsAssignmentWrapperImpl.regionsStateMap;
locked
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496095#comment-16496095
]
liubangchen commented on HBASE-20654:
-
When Hbase as cloud product,we need gather many metrics by jmx . regions state
info is very important metrics . If expose these regions state info to jxm is
very useful.
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch, HBASE-20654-2.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495492#comment-16495492
]
Sean Busbey commented on HBASE-20654:
-
(ClusterMetrics.getRegionStatesInTransition is IA.Private though)
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch, HBASE-20654-2.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495493#comment-16495493
]
stack commented on HBASE-20654:
---
bq. Let me get better summary when the utilization of this enhancement is
embodied in more concrete use cases.
I'd prefer that we first explain why we need something first before we commit
it.
How is this view to be used? Why is it outside of the realm of hbck?
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch, HBASE-20654-2.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495488#comment-16495488
]
Sean Busbey commented on HBASE-20654:
-
you can also get this in the Java API from the ClusterMetrics object returned
from Admin, I think.
In the shell it'd look like:
{code}
hbase(main):038:0> regions_in_transition = @hbase.admin.instance_eval do
hbase(main):039:1* @admin.get_cluster_metrics.get_region_states_in_transition
hbase(main):040:1> end
=> #
{code}
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch, HBASE-20654-2.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495475#comment-16495475
]
Ted Yu commented on HBASE-20654:
Let me get better summary when the utilization of this enhancement is embodied
in more concrete use cases.
I understand that web UI and logs contain regions in transition. Such
information currently is for manual analysis.
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch, HBASE-20654-2.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495444#comment-16495444
]
stack commented on HBASE-20654:
---
bq. Knowing regions in transition programmatically gives more choices to shell
scripting.
What are these shell scripts going to do? How does this relate to hbck if at
all?
bq. Please let me know these channels, other than the Web UI.
webui and logs... hbck knows whats RIT.
What are you trying to do that requires this jmx view? That'd help.
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch, HBASE-20654-2.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495402#comment-16495402
]
Ted Yu commented on HBASE-20654:
Several shell commands require region name as parameter. Knowing regions in
transition programmatically gives more choices to shell scripting.
bq. The list exposed here is available via other channels
Please let me know these channels, other than the Web UI.
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch, HBASE-20654-2.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495394#comment-16495394
]
stack commented on HBASE-20654:
---
bq. This is useful for troubleshooting region assignment related issues.
How?
The list exposed here is available via other channels. Why exposing via JMX too?
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch, HBASE-20654-2.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495369#comment-16495369
]
Ted Yu commented on HBASE-20654:
With this additional change:
https://pastebin.com/EhvpwhY7
I was able to get TestRecoverMetaProcedure to pass.
{code}
+if (metricsRecordBuilder != null) {
+
+ metricsAssignmentWrapper.snapshot();
{code}
Please check that metricsAssignmentWrapper is not null before calling snapshot.
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch, HBASE-20654-2.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495332#comment-16495332
]
Ted Yu commented on HBASE-20654:
This utility provides programmatic way for operator to obtain the regions in
transition.
This is useful for troubleshooting region assignment related issues.
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch, HBASE-20654-2.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495315#comment-16495315
]
stack commented on HBASE-20654:
---
What is the use-case for a (truncated) hbase:meta table view via jmx? Thanks.
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch, HBASE-20654-2.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495269#comment-16495269
]
Ted Yu commented on HBASE-20654:
Test failure is reproducible:
{code}
testPrepareWithNoCluster(org.apache.hadoop.hbase.master.procedure.TestRecoverMetaProcedure)
Time elapsed: 0.042 sec <<< ERROR!
org.apache.hadoop.metrics2.MetricsException: Metrics source
Master,sub=AssignmentManager already exists!
at
org.apache.hadoop.hbase.master.procedure.TestRecoverMetaProcedure.testPrepareWithNoCluster(TestRecoverMetaProcedure.java:99)
{code}
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch, HBASE-20654-2.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495114#comment-16495114
]
Hadoop QA commented on HBASE-20654:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
24s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} hbase-hadoop2-compat: The patch generated 1 new + 4
unchanged - 1 fixed = 5 total (was 5) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
13s{color} | {color:red} hbase-server: The patch generated 5 new + 38 unchanged
- 0 fixed = 43 total (was 38) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
58s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 53s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
18s{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed =
1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
25s{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
30s{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 40s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 76m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Inconsistent synchronization of
org.apache.hadoop.hbase.master.assignment.MetricsAssignmentWrapperImpl.regionsStateMap;
locked 50% of time Unsynchronized
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495057#comment-16495057
]
liubangchen commented on HBASE-20654:
-
I've already attached the latest patch . Wait for QA to run.
Thank you very much.
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch, HBASE-20654-2.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494912#comment-16494912
]
Ted Yu commented on HBASE-20654:
Can you attach the latest patch for QA run ?
Thanks
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494865#comment-16494865
]
liubangchen commented on HBASE-20654:
-
This is my first independent submission of patchs, and I need your guidance.
The exported jmx example is as follows：
{code:json}
{
"beans" : [ {
"name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
"modelerType" : "Master,sub=AssignmentManager",
"tag.regionsOffline" : "default_t2_1b48dede1f20e6d4a61g08499d3sa52i",
"tag.regionsOpening" : "",
"tag.regionsOpen" : "",
"tag.regionsClosing" : "",
"tag.regionsClosed" : "default_t1_1b48dede1f20e6d4a61308499b3ba523",
"tag.regionsSplitting" : "",
"tag.regionsSplit" : "",
"tag.regionsFailedOpen" : "",
"tag.regionsFailedClose" : "",
"tag.regionsMerging" : "",
"tag.regionsMerged" : "",
"tag.regionsSplittingNew" : "",
"tag.Context" : "master",
"tag.Hostname" : "LIUBANGCHEN-MB1",
"operationCount" : 2,
"ritOldestAge" : 300341,
"RitDuration_num_ops" : 0,
"RitDuration_min" : 0,
"RitDuration_max" : 0,
"RitDuration_mean" : 0,
"RitDuration_25th_percentile" : 0,
"RitDuration_median" : 0,
"RitDuration_75th_percentile" : 0,
"RitDuration_90th_percentile" : 0,
"RitDuration_95th_percentile" : 0,
"RitDuration_98th_percentile" : 0,
"RitDuration_99th_percentile" : 0,
"RitDuration_99.9th_percentile" : 0,
"ritCountOverThreshold" : 1,
"ritCount" : 1
} ]
}
{code}
qyr=Hadoop:service=HBase,name=Master,sub=AssignmentManager
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494655#comment-16494655
]
Ted Yu commented on HBASE-20654:
Patch v3 only contained new classes.
Please combine v2 and v3 into v4, after addressing review comments.
If possible, please collect output from /jmx of master UI and attach relevant
snippet to this JIRA.
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494641#comment-16494641
]
Ted Yu commented on HBASE-20654:
Looks like patch v2 didn't include MetricsAssignmentManagerSourceImpl since it
is new class.
Use this command and pipe its output to the end of next patch:
git diff --cached
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494638#comment-16494638
]
Ted Yu commented on HBASE-20654:
Can you take a look at the following compilation error based on patch v2 ?
{code}
[ERROR]
/Users/tyu/master/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsAssignmentManagerSourceImpl.java:[48,13]
cannot find symbol
symbol: class MetricsAssignmentWrapper
location: class
org.apache.hadoop.hbase.master.MetricsAssignmentManagerSourceImpl
[ERROR]
/Users/tyu/master/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsAssignmentManagerSourceImpl.java:[50,47]
cannot find symbol
symbol: class MetricsAssignmentWrapper
location: class
org.apache.hadoop.hbase.master.MetricsAssignmentManagerSourceImpl
{code}
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494633#comment-16494633
]
liubangchen commented on HBASE-20654:
-
Code review link : [https://reviews.apache.org/r/67376/diff/1#index_header]
Under your suggestions, I modified the code.Could you help me review it?
Thanks.
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494625#comment-16494625
]
Ted Yu commented on HBASE-20654:
Please upload next patch to https://reviews.apache.org/
Repository: hbase-git
Groups: hbase
Bugs: HBASE-20654
Thanks
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493665#comment-16493665
]
Ted Yu commented on HBASE-20654:
{code}
+
metricsRecordBuilder.tag(Interns.info(REGIONS_STATE_FAILED_OPEN,REGIONS_STATE_FAILED_OPEN_DESC),metricsAssignmentWrapper.getFailedOpenRegions());
{code}
Please limit line length to 100 characters.
{code}
+//assignmentManagerSource = CompatibilitySingletonFactory.getInstance(
+//MetricsAssignmentManagerSource.class);
{code}
Drop commented out code.
{code}
+ public static final String
METRIC_MAXNUM_REGIONS_IN_EACH_STATE="hbase.assignment.metric.maxmum.regions.in.each.state";
typo: maxnum -> maximum. If you meant maximum number of regions, use "max.num"
instead.
{code}
+ private static final int DEFAULT_METRIC_MAXNUM_REGIONS_IN_EACH_STATE = 200;
{code}
Consider lowering the default count since the count is per state. How about
using 100 as the default ?
{code}
+ protected int getMaxnumRegionInEachState(){
{code}
num -> Num
Region -> Regions
{code}
+return StringUtils.join(regions, ",");
{code}
I think using ', ' as delimiter would make the output more readable.
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493558#comment-16493558
]
Ted Yu commented on HBASE-20654:
{code}
+datalist.addAll(metalist);
{code}
Does the above put the meta regions at the end of the datalist ? meta regions
should be placed at the beginning.
getRegionsListByState is called by various getters. So it should be optimized.
It can return a Map . getMetrics() calls the method
once and iterates the result, calls {{metricsRecordBuilder.tag}} for each entry.
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493539#comment-16493539
]
Ted Yu commented on HBASE-20654:
{code}
+ String REGIONS_STATE_OFFLINE="regionsOffline";
{code}
You can globally replace REGIONS_STATE_ with REGIONS_ (offline implies a state).
Looking at MetricsAssignmentWrapper.java, it seems to be the diff based on one
other commit. Please generate as new file.
After 'git add', I normally use the following command:
{code}
git diff --cached
{code}
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493519#comment-16493519
]
Hadoop QA commented on HBASE-20654:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HBASE-20654 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.7.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-20654 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12925545/HBASE-20654-1.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/12994/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493514#comment-16493514
]
liubangchen commented on HBASE-20654:
-
Expose metrics to tag of Hadoop:service=HBase,name=Master,sub=AssignmentManager:
{code:json}
"beans" : [ {
"name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
"modelerType" : "Master,sub=AssignmentManager",
"tag.regionsOffline" :
"Namespace_tablename_regionname,Namespace_tablename_regionname",
"tag.regionsOpening" : "",
"tag.regionsOpen" : "",
"tag.regionsClosing" : "",
"tag.regionsClosed" : "",
"tag.regionsSplitting" : "",
"tag.regionsSplit" : "",
"tag.regionsFailedOpen" : "",
"tag.regionsFailedClose" : "",
"tag.regionsMerging" : "",
"tag.regionsMerged" : "",
"tag.regionsSplittingNew" : "",
"ritCount" : 0
} ]
}
{code}
1. Add metrics tags, add attributes to interface MetricsAssignmentManagerSource
{code:java}
String REGIONS_STATE_OFFLINE="regionsOffline";
String REGIONS_STATE_OFFLINE_DESC="Regions with state OFFLINE";
String REGIONS_STATE_OPENING="regionsOpening";
String REGIONS_STATE_OPENING_DESC="Regions with state OPENING";
String REGIONS_STATE_OPEN="regionsOpen";
String REGIONS_STATE_OPEN_DESC="Regions with state OPEN";
String REGIONS_STATE_CLOSING="regionsClosing";
String REGIONS_STATE_CLOSING_DESC="Regions with state CLOSING";
String REGIONS_STATE_CLOSED="regionsClosed";
String REGIONS_STATE_CLOSED_DESC="Regions with state CLOSED";
String REGIONS_STATE_SPLITTING="regionsSplitting";
String REGIONS_STATE_SPLITTING_DESC="Regions with state SPLITTING";
String REGIONS_STATE_SPLIT="regionsSplit";
String REGIONS_STATE_SPLIT_DESC="Regions with state SPLIT";
String REGIONS_STATE_FAILED_OPEN="regionsFailedOpen";
String REGIONS_STATE_FAILED_OPEN_DESC="Regions with state failed open";
String REGIONS_STATE_FAILED_CLOSE="regionsFailedClose";
String REGIONS_STATE_FAILED_CLOSE_DESC="Regions with state failed close";
String REGIONS_STATE_MERGING="regionsMerging";
String REGIONS_STATE_MERGING_DESC="Regions with state merging";
String REGIONS_STATE_MERGED="regionsMerged";
String REGIONS_STATE_MERGED_DESC="Regions with state merged";
String REGIONS_STATE_SPLITTING_NEW="regionsSplittingNew";
String REGIONS_STATE_SPLITTING_NEW_DESC="Regions with state";
{code}
2. Modify class MetricsAssignmentManagerSourceImpl
{code:java}
@Override
public void getMetrics(MetricsCollector metricsCollector, boolean all) {
MetricsRecordBuilder metricsRecordBuilder =
metricsCollector.addRecord(metricsName);
if(metricsRecordBuilder!=null){
metricsRecordBuilder.tag(Interns.info(REGIONS_STATE_OFFLINE,REGIONS_STATE_OFFLINE_DESC),metricsAssignmentWrapper.getOfflineRegions());
metricsRegistry.snapshot(metricsRecordBuilder, all);
}
{code}
3. Add new config is to control the number of region names in each region
state. How about naming the config:
hbase.assignment.metric.maxmum.regions.in.each.state
4. Get metrics by class MetricsAssignmentWrapperImpl
{code:java}
private String getRegionsListByState(RegionState.State state) {
List regions = new LinkedList();
RegionStates rstates = this.am.getRegionStates();
if (rstates == null) {
return "";
}
List list =
rstates.getRegionsInTransition();
if (list == null || list.isEmpty()) {
return "";
}
List metalist = new
LinkedList();
List datalist = new
LinkedList();
int i = 0;
for (RegionStates.RegionStateNode rsn : list) {
if (rsn.getTable().isSystemTable()) {
metalist.add(rsn);
} else {
datalist.add(rsn);
}
}
datalist.addAll(metalist);
for (RegionStates.RegionStateNode rsn : datalist) {
if (rsn.getState().equals(state)) {
regions.add(String.format("%s_%s_%s",
rsn.getTable().getNamespaceAsString(), rsn.getTable().getNameAsString(),
rsn.getRegionInfo() == null ? "" : rsn.getRegionInfo().getEncodedName()));
i++;
}
if (i > this.am.getMaxnumRegionInEachState()) {
break;
}
}
return StringUtils.join(regions, ",");
}
{code}
[~yuzhih...@gmail.com] Can you give me some suggestions? Let me submit a
patch. you can help me review it. thanks.
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492698#comment-16492698
]
Ted Yu commented on HBASE-20654:
bq. Modify the updateRegionsInTransitionMetrics method of class
AssignmentManager and add the parameter name
The new config is to control the number of region names in each region state.
How about naming the config:
hbase.assignment.metric.maxmum.regions.in.each.state
You can retrieve the value for this new config in:
{code}
public AssignmentManager(final MasterServices master, final RegionStateStore
stateStore) {
{code}
and save the value in an int field of AssignmentManager.
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492693#comment-16492693
]
Ted Yu commented on HBASE-20654:
Please refer to the following code in MetricsMasterSourceImpl :
{code}
.tag(Interns.info(DEAD_REGION_SERVERS_NAME, DEAD_REGION_SERVERS_DESC),
masterWrapper.getDeadRegionServers())
{code}
I think exposing each region in transition would be difficult since we don't
know the region names at startup.
We can follow existing metric such as the following:
{code}
"name" : "Hadoop:service=HBase,name=Master,sub=Server",
"modelerType" : "Master,sub=Server",
"tag.liveRegionServers" :
"ctr-e138-1518143905142-329221-01-02.hwx.site,16020,1527267274818;ctr-e138-1518143905142-329221-01-03.hwx.site,16020,1527267276442;ctr-e138-1518143905142-329221-01-05.hwx.site,16020,1527267274409;ctr-e138-1518143905142-329221-01-06.hwx.site,16020,1527267273345;ctr-e138-1518143905142-329221-01-07.hwx.site,16020,1527267275021",
"tag.deadRegionServers" : "",
{code}
Meaning, create tag for each region state: OPENING, CLOSED, etc. For each tag,
list the top N (configurable) encoded region names.
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492669#comment-16492669
]
liubangchen commented on HBASE-20654:
-
expose metrics like this:
{code:json}
{
"name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManger",
"modelerType" : "Master,sub=AssignmentManger",
"tag.Context" : "master",
"tag.Hostname" : "10.59.168.84",
"ritOldestAge" : 0,
"ritCount" : 0,
"tag.RegionState.Namespace_Tablename_Regiona":" "OFFLINE",
"tag.RegionState.Namespace_Tablename_Regionb":" "OPEN",
"tag.RegionState.Namespace_Tablename_Regionc":" "CLOSED",
...
{code}
Code implementation:
1. Add a method to class MetricsAssignmentManagerSource
{code:java}
void updateRegionState(String regionname,String state);
{code}
2. Add a method to class MetricsAssignmentManagerSourceImpl
{code:java}
@Override
public void updateRegionState(String regionname,String state){
metricsRegistry.tag(regionname,"",state,true);
}
{code}
3. Add a method to class MetricsAssignmentManager
{code:java}
public void updateRegionState(String regionname,String state){
assignmentManagerSource.updateRegionState(regionname,state);
}
{code}
4. Modify the updateRegionsInTransitionMetrics method of class
AssignmentManager and add the parameter name
hbase.assignment.metric.regionstate.maxmum.expose to control the maximum number
of exports
[~yuzhih...@gmail.com] What's your opinion?
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
