[jira] [Commented] (HBASE-5061) StoreFileLocalityChecker
[ https://issues.apache.org/jira/browse/HBASE-5061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13408314#comment-13408314 ] Elliott Clark commented on HBASE-5061: -- The o.a.h.h.master.balance.RegionLocationFinder does something like this. It would be cool if functionality could be merged. Maybe after this goes in I'll open a ticket to do that work, since it looks like this is more complete than the balancer version. StoreFileLocalityChecker Key: HBASE-5061 URL: https://issues.apache.org/jira/browse/HBASE-5061 Project: HBase Issue Type: New Feature Affects Versions: 0.96.0, 0.94.1 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Attachments: HBASE-5061-0.94.patch, HBASE-5061-0.94.patch, HBASE-5061.patch, HBASE-5061.patch org.apache.hadoop.hbase.HFileLocalityChecker [options] A tool to report the number of local and nonlocal HFile blocks, and the ratio of as a percentage. Where options are: |-f file|Analyze a store file| |-r region|Analyze all store files for the region| |-t table|Analyze all store files for regions of the table served by the local regionserver| |-h host|Consider host local, defaults to the local host| |-v|Verbose operation| -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5061) StoreFileLocalityChecker
[ https://issues.apache.org/jira/browse/HBASE-5061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13408379#comment-13408379 ] Andrew Purtell commented on HBASE-5061: --- bq. The o.a.h.h.master.balance.RegionLocationFinder does something like this. It would be cool if functionality could be merged. Maybe after this goes in I'll open a ticket to do that work Sounds good to me. StoreFileLocalityChecker Key: HBASE-5061 URL: https://issues.apache.org/jira/browse/HBASE-5061 Project: HBase Issue Type: New Feature Affects Versions: 0.96.0, 0.94.1 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Attachments: HBASE-5061-0.94.patch, HBASE-5061-0.94.patch, HBASE-5061.patch, HBASE-5061.patch org.apache.hadoop.hbase.HFileLocalityChecker [options] A tool to report the number of local and nonlocal HFile blocks, and the ratio of as a percentage. Where options are: |-f file|Analyze a store file| |-r region|Analyze all store files for the region| |-t table|Analyze all store files for regions of the table served by the local regionserver| |-h host|Consider host local, defaults to the local host| |-v|Verbose operation| -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5061) StoreFileLocalityChecker
[
https://issues.apache.org/jira/browse/HBASE-5061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13408410#comment-13408410
]
Zhihong Ted Yu commented on HBASE-5061:
---
Nice work.
{code}
+public class StoreFileLocalityChecker {
{code}
Should @InterfaceAudience.Private be added to the class ?
{code}
+void reportTotals(double min, double max, double average) throws
IOException;
{code}
Since total isn't included in the report, maybe there is a better name for the
above method ?
{code}
+ private PairPath,HDFSBlocksDistribution getBlocksForPath(Path path)
{code}
Space after comma.
{code}
+ if (familyDir.getPath().toString().endsWith(.tmp)) {
+continue;
{code}
Is there other directory that should be skipped ?
{code}
+// Is the region assigned to the desired server?
+byte[] value = row.getValue(HConstants.CATALOG_FAMILY,
HConstants.SERVER_QUALIFIER);
+if (value == null || value.length serverName.length) {
+ return true;
+}
{code}
If serverName is a prefix of value, the region would be included in the
collection returned. Is that intended ?
{code}
+ static class SimpleReporter implements Reporter {
{code}
The above class can be private, right ?
Same with JSONReporter class.
StoreFileLocalityChecker
Key: HBASE-5061
URL: https://issues.apache.org/jira/browse/HBASE-5061
Project: HBase
Issue Type: New Feature
Affects Versions: 0.96.0, 0.94.1
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
Attachments: HBASE-5061-0.94.patch, HBASE-5061-0.94.patch,
HBASE-5061.patch, HBASE-5061.patch
org.apache.hadoop.hbase.HFileLocalityChecker [options]
A tool to report the number of local and nonlocal HFile blocks, and the ratio
of as a percentage.
Where options are:
|-f file|Analyze a store file|
|-r region|Analyze all store files for the region|
|-t table|Analyze all store files for regions of the table served by the
local regionserver|
|-h host|Consider host local, defaults to the local host|
|-v|Verbose operation|
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5061) StoreFileLocalityChecker
[
https://issues.apache.org/jira/browse/HBASE-5061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13408443#comment-13408443
]
Andrew Purtell commented on HBASE-5061:
---
Thanks for the review Ted.
bq. Should @InterfaceAudience.Private be added to the class ?
Ok
bq. Is there other directory that should be skipped ?
Not previously but this is pending live cluster testing.
bq. Since total isn't included in the report, maybe there is a better name for
the above method ?
Ok, let's call it {{reportStats}}
bq. If serverName is a prefix of value, the region would be included in the
collection returned. Is that intended ?
Yes, so short names can be used in place of FQDN.
bq. The above class can be private, right ?
Sure.
StoreFileLocalityChecker
Key: HBASE-5061
URL: https://issues.apache.org/jira/browse/HBASE-5061
Project: HBase
Issue Type: New Feature
Affects Versions: 0.96.0, 0.94.1
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
Attachments: HBASE-5061-0.94.patch, HBASE-5061-0.94.patch,
HBASE-5061.patch, HBASE-5061.patch
org.apache.hadoop.hbase.HFileLocalityChecker [options]
A tool to report the number of local and nonlocal HFile blocks, and the ratio
of as a percentage.
Where options are:
|-f file|Analyze a store file|
|-r region|Analyze all store files for the region|
|-t table|Analyze all store files for regions of the table served by the
local regionserver|
|-h host|Consider host local, defaults to the local host|
|-v|Verbose operation|
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5061) StoreFileLocalityChecker
[ https://issues.apache.org/jira/browse/HBASE-5061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13408522#comment-13408522 ] Hadoop QA commented on HBASE-5061: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12535469/HBASE-5061-0.94.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.replication.TestReplication org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol org.apache.hadoop.hbase.master.TestSplitLogManager Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2338//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2338//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2338//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2338//console This message is automatically generated. StoreFileLocalityChecker Key: HBASE-5061 URL: https://issues.apache.org/jira/browse/HBASE-5061 Project: HBase Issue Type: New Feature Affects Versions: 0.96.0, 0.94.1 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Attachments: HBASE-5061-0.94.patch, HBASE-5061-0.94.patch, HBASE-5061-0.94.patch, HBASE-5061.patch, HBASE-5061.patch, HBASE-5061.patch org.apache.hadoop.hbase.HFileLocalityChecker [options] A tool to report the number of local and nonlocal HFile blocks, and the ratio of as a percentage. Where options are: |-f file|Analyze a store file| |-r region|Analyze all store files for the region| |-t table|Analyze all store files for regions of the table served by the local regionserver| |-h host|Consider host local, defaults to the local host| |-v|Verbose operation| -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5061) StoreFileLocalityChecker
[ https://issues.apache.org/jira/browse/HBASE-5061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13408559#comment-13408559 ] Hadoop QA commented on HBASE-5061: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12535469/HBASE-5061-0.94.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2340//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2340//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2340//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2340//console This message is automatically generated. StoreFileLocalityChecker Key: HBASE-5061 URL: https://issues.apache.org/jira/browse/HBASE-5061 Project: HBase Issue Type: New Feature Affects Versions: 0.96.0, 0.94.1 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Attachments: HBASE-5061-0.94.patch, HBASE-5061-0.94.patch, HBASE-5061-0.94.patch, HBASE-5061.patch, HBASE-5061.patch, HBASE-5061.patch org.apache.hadoop.hbase.HFileLocalityChecker [options] A tool to report the number of local and nonlocal HFile blocks, and the ratio of as a percentage. Where options are: |-f file|Analyze a store file| |-r region|Analyze all store files for the region| |-t table|Analyze all store files for regions of the table served by the local regionserver| |-h host|Consider host local, defaults to the local host| |-v|Verbose operation| -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5061) StoreFileLocalityChecker
[ https://issues.apache.org/jira/browse/HBASE-5061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13401571#comment-13401571 ] Andrew Purtell commented on HBASE-5061: --- Updated patch adds '-j' option to produce JSON output. StoreFileLocalityChecker Key: HBASE-5061 URL: https://issues.apache.org/jira/browse/HBASE-5061 Project: HBase Issue Type: New Feature Affects Versions: 0.96.0, 0.94.1 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Attachments: HBASE-5061-0.94.patch, HBASE-5061.patch org.apache.hadoop.hbase.HFileLocalityChecker [options] A tool to report the number of local and nonlocal HFile blocks, and the ratio of as a percentage. Where options are: |-f file|Analyze a store file| |-r region|Analyze all store files for the region| |-t table|Analyze all store files for regions of the table served by the local regionserver| |-h host|Consider host local, defaults to the local host| |-v|Verbose operation| -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5061) StoreFileLocalityChecker
[ https://issues.apache.org/jira/browse/HBASE-5061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13401612#comment-13401612 ] Hadoop QA commented on HBASE-5061: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12533510/HBASE-5061-0.94.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2262//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2262//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2262//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2262//console This message is automatically generated. StoreFileLocalityChecker Key: HBASE-5061 URL: https://issues.apache.org/jira/browse/HBASE-5061 Project: HBase Issue Type: New Feature Affects Versions: 0.96.0, 0.94.1 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Attachments: HBASE-5061-0.94.patch, HBASE-5061.patch org.apache.hadoop.hbase.HFileLocalityChecker [options] A tool to report the number of local and nonlocal HFile blocks, and the ratio of as a percentage. Where options are: |-f file|Analyze a store file| |-r region|Analyze all store files for the region| |-t table|Analyze all store files for regions of the table served by the local regionserver| |-h host|Consider host local, defaults to the local host| |-v|Verbose operation| -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5061) StoreFileLocalityChecker
[ https://issues.apache.org/jira/browse/HBASE-5061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13176311#comment-13176311 ] Lars Hofhansl commented on HBASE-5061: -- From looking at the code, this would produce data locality with respect to certain host (if I understand this correctly). Wouldn't we want to report locality from the viewpoint the respective regionserver(s)? I.e. regions for a table might be on many regionservers, but for each the data might be local to the regionserver. StoreFileLocalityChecker Key: HBASE-5061 URL: https://issues.apache.org/jira/browse/HBASE-5061 Project: HBase Issue Type: New Feature Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Attachments: StoreFileLocalityChecker.java org.apache.hadoop.hbase.HFileLocalityChecker [options] A tool to report the number of local and nonlocal HFile blocks, and the ratio of as a percentage. Where options are: |-f file|Analyze a store file| |-r region|Analyze all store files for the region| |-t table|Analyze all store files for regions of the table served by the local regionserver| |-h host|Consider host local, defaults to the local host| |-v|Verbose operation| -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5061) StoreFileLocalityChecker
[ https://issues.apache.org/jira/browse/HBASE-5061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13176322#comment-13176322 ] Andrew Purtell commented on HBASE-5061: --- bq. From looking at the code, this would produce data locality with respect to certain host (if I understand this correctly). Yes this is the intent, to report locality to a given regionserver host, by default the local host as determined by a HBase style reverse lookup, assuming ops would run it either on an ops box (or the master) and supply the desired local host name via the '-h' option, or local to the RS. I was thinking one use case could be to iterate over each cluster node and trigger major compactions for region(s) depending on the output of this tool. Of you are looking for a whole cluster report? StoreFileLocalityChecker Key: HBASE-5061 URL: https://issues.apache.org/jira/browse/HBASE-5061 Project: HBase Issue Type: New Feature Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Attachments: StoreFileLocalityChecker.java org.apache.hadoop.hbase.HFileLocalityChecker [options] A tool to report the number of local and nonlocal HFile blocks, and the ratio of as a percentage. Where options are: |-f file|Analyze a store file| |-r region|Analyze all store files for the region| |-t table|Analyze all store files for regions of the table served by the local regionserver| |-h host|Consider host local, defaults to the local host| |-v|Verbose operation| -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5061) StoreFileLocalityChecker
[ https://issues.apache.org/jira/browse/HBASE-5061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13176326#comment-13176326 ] Andrew Purtell commented on HBASE-5061: --- So, currently this tool is meant to check the locality of files or regions to a given single host, however it could be modified such that if the '-h' option is supplied then the report looks only at the given host, otherwise it will enumerate the live hosts in ClusterStatus and return results for all. While at it, a new option '-j' for reporting in JSON. Is this more what you have in mind Lars? StoreFileLocalityChecker Key: HBASE-5061 URL: https://issues.apache.org/jira/browse/HBASE-5061 Project: HBase Issue Type: New Feature Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Attachments: StoreFileLocalityChecker.java org.apache.hadoop.hbase.HFileLocalityChecker [options] A tool to report the number of local and nonlocal HFile blocks, and the ratio of as a percentage. Where options are: |-f file|Analyze a store file| |-r region|Analyze all store files for the region| |-t table|Analyze all store files for regions of the table served by the local regionserver| |-h host|Consider host local, defaults to the local host| |-v|Verbose operation| -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5061) StoreFileLocalityChecker
[ https://issues.apache.org/jira/browse/HBASE-5061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13176332#comment-13176332 ] Lars Hofhansl commented on HBASE-5061: -- That's what I had envisioned for the -t option at least. Would be nice to have a report on all regions of a table. But maybe that would not be useful as the number of region servers grows...? +1 on JSON output. StoreFileLocalityChecker Key: HBASE-5061 URL: https://issues.apache.org/jira/browse/HBASE-5061 Project: HBase Issue Type: New Feature Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Attachments: StoreFileLocalityChecker.java org.apache.hadoop.hbase.HFileLocalityChecker [options] A tool to report the number of local and nonlocal HFile blocks, and the ratio of as a percentage. Where options are: |-f file|Analyze a store file| |-r region|Analyze all store files for the region| |-t table|Analyze all store files for regions of the table served by the local regionserver| |-h host|Consider host local, defaults to the local host| |-v|Verbose operation| -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5061) StoreFileLocalityChecker
[ https://issues.apache.org/jira/browse/HBASE-5061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13176091#comment-13176091 ] Andrew Purtell commented on HBASE-5061: --- @Weidong, yes this uses the utility functions done for HBASE-4114. StoreFileLocalityChecker Key: HBASE-5061 URL: https://issues.apache.org/jira/browse/HBASE-5061 Project: HBase Issue Type: New Feature Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Attachments: StoreFileLocalityChecker.java org.apache.hadoop.hbase.HFileLocalityChecker [options] A tool to report the number of local and nonlocal HFile blocks, and the ratio of as a percentage. Where options are: |-f file|Analyze a store file| |-r region|Analyze all store files for the region| |-t table|Analyze all store files for regions of the table served by the local regionserver| |-h host|Consider host local, defaults to the local host| |-v|Verbose operation| -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
