[hbase] annotated tag rel/2.2.7 created (now 27df11c)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to annotated tag rel/2.2.7 in repository https://gitbox.apache.org/repos/asf/hbase.git. at 27df11c (tag) tagging 0fc18a9056e5eb3a80fdbde916865607946c5195 (commit) replaces rel/2.2.6 by Guanghao Zhang on Mon Apr 19 16:06:28 2021 +0800 - Log - HBase 2.2.7 -BEGIN PGP SIGNATURE- iQIzBAABCAAdFiEEqgCu7gDR/ij/88WIApVEXRfxNG4FAmB9OgUACgkQApVEXRfx NG7NgQ//YUGyLhNQJB7g9/1frV3NA3L8Pm1jk5GWNw5+gCzOxChXgRy4VxcMVoD2 aTcEyATCKj1WIQ6I1L0fiAW8fErUN1WTlknT2u6W6jtVn0mj6LYyY9V3cdEPte3g q+jAgaClVEYtOIiATO7YFY/kLHLvaurLpl3GX5+xu0KpKCtVrkOeDkgvQRN+bQIe aS6wPxUJrDiwwbbp6v4H8xZz4ujp4LpE4rtkNZqWTE2ylMx9Hdua1Cvug2LheJVd 8uBQWAGHtGyarmmzuBZx6tWkPNAKH3Ghu4C/nobHCyqeqqd6gK8fJsnyyh5VhaUs bvD4zFPwPDPpbDimihcMxUawHUpMK5Ebnac6lYqL0qM6rzVFRvShtW1dwepW6Zer UhHYAqgiPj5Lv09ShOX7H55KwMtNfrEigBmC8mFInMMi7KJDsV7/rq7oDRaezWeh sn+qOJ1eIy9fSIpFxnyX5eOf5sxfMZpkFHZPAZm8UNQVCgZMkyI1NfB+ImRzNmaK E2hdLLW7GbGgYq7yz9elb0/xyIafe1/D42I5fD9MsNlsdFyFm/5YZDYDusVCYfry Yl8UH1sIhYMOOVUyuTQeESro42FtIzl5kRRR2KGPM9OqF11gSPEPAaivFiRKk8U0 t6w/+L5UyN38HGITJIPjbnrP9tVEG59AjdIP5kgdDk5xRPN1yig= =64y3 -END PGP SIGNATURE- --- No new revisions were added by this update.
[hbase] branch master updated (bf78246 -> 94f4479)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hbase.git. from bf78246 HBASE-25775 Use a special balancer to deal with maintenance mode (#3161) add 94f4479 HBASE-25780 Add 2.2.7 to download page (#3175) No new revisions were added by this update. Summary of changes: src/site/xdoc/downloads.xml | 16 1 file changed, 8 insertions(+), 8 deletions(-)
svn commit: r47113 [1/4] - /release/hbase/2.2.7/
Author: zghao Date: Fri Apr 16 05:07:26 2021 New Revision: 47113 Log: Apache HBase 2.2.7 Added: release/hbase/2.2.7/ release/hbase/2.2.7/CHANGES.md release/hbase/2.2.7/RELEASENOTES.md release/hbase/2.2.7/api_compare_2.2.7RC0_to_2.2.6.html release/hbase/2.2.7/hbase-2.2.7-bin.tar.gz (with props) release/hbase/2.2.7/hbase-2.2.7-bin.tar.gz.asc release/hbase/2.2.7/hbase-2.2.7-bin.tar.gz.sha512 release/hbase/2.2.7/hbase-2.2.7-client-bin.tar.gz (with props) release/hbase/2.2.7/hbase-2.2.7-client-bin.tar.gz.asc release/hbase/2.2.7/hbase-2.2.7-client-bin.tar.gz.sha512 release/hbase/2.2.7/hbase-2.2.7-src.tar.gz (with props) release/hbase/2.2.7/hbase-2.2.7-src.tar.gz.asc release/hbase/2.2.7/hbase-2.2.7-src.tar.gz.sha512
svn commit: r47113 [2/4] - /release/hbase/2.2.7/
Added: release/hbase/2.2.7/CHANGES.md == --- release/hbase/2.2.7/CHANGES.md (added) +++ release/hbase/2.2.7/CHANGES.md Fri Apr 16 05:07:26 2021 @@ -0,0 +1,1721 @@ +# HBASE Changelog + + +## Release 2.2.7 - Unreleased (as of 2021-04-11) + + + +### NEW FEATURES: + +| JIRA | Summary | Priority | Component | +|: |: | :--- |: | +| [HBASE-25587](https://issues.apache.org/jira/browse/HBASE-25587) | [hbck2] Schedule SCP for all unknown servers | Major | hbase-operator-tools, hbck2 | +| [HBASE-25460](https://issues.apache.org/jira/browse/HBASE-25460) | Expose drainingServers as cluster metric | Major | metrics | +| [HBASE-24776](https://issues.apache.org/jira/browse/HBASE-24776) | [hbtop] Support Batch mode | Major | hbtop | + + +### IMPROVEMENTS: + +| JIRA | Summary | Priority | Component | +|: |: | :--- |: | +| [HBASE-25656](https://issues.apache.org/jira/browse/HBASE-25656) | Backport to branch-2.2: [HBASE-25548 Optionally allow snapshots to preserve cluster's max filesize config by setting it into table descriptor] | Major | . | +| [HBASE-25621](https://issues.apache.org/jira/browse/HBASE-25621) | Balancer should check region plan source to avoid misplace region groups | Major | Balancer | +| [HBASE-25541](https://issues.apache.org/jira/browse/HBASE-25541) | In WALEntryStream, set the current path to null while dequeing the log | Major | . | +| [HBASE-25536](https://issues.apache.org/jira/browse/HBASE-25536) | Remove 0 length wal file from logQueue if it belongs to old sources. | Major | Replication | +| [HBASE-25329](https://issues.apache.org/jira/browse/HBASE-25329) | Dump region hashes in logs for the regions that are stuck in transition for more than a configured amount of time | Minor | . | +| [HBASE-25475](https://issues.apache.org/jira/browse/HBASE-25475) | Improve unit test for HBASE-25445 : SplitWALRemoteProcedure failed to archive split WAL | Minor | wal | +| [HBASE-25449](https://issues.apache.org/jira/browse/HBASE-25449) | 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml | Major | conf | +| [HBASE-25483](https://issues.apache.org/jira/browse/HBASE-25483) | set the loadMeta log level to debug. | Major | MTTR, Region Assignment | +| [HBASE-25318](https://issues.apache.org/jira/browse/HBASE-25318) | Configure where IntegrationTestImportTsv generates HFiles | Minor | integration tests | +| [HBASE-24850](https://issues.apache.org/jira/browse/HBASE-24850) | CellComparator perf improvement | Critical | Performance, scan | +| [HBASE-25425](https://issues.apache.org/jira/browse/HBASE-25425) | Some notes on RawCell | Trivial | . | +| [HBASE-25246](https://issues.apache.org/jira/browse/HBASE-25246) | Backup/Restore hbase cell tags. | Major | backup&restore | +| [HBASE-25328](https://issues.apache.org/jira/browse/HBASE-25328) | Add builder method to create Tags. | Minor | . | +| [HBASE-25339](https://issues.apache.org/jira/browse/HBASE-25339) | Method parameter and member variable are duplicated in checkSplittable() of SplitTableRegionProcedure | Minor | . | +| [HBASE-25237](https://issues.apache.org/jira/browse/HBASE-25237) | 'hbase master stop' shuts down the cluster, not the master only | Major | . | +| [HBASE-25323](https://issues.apache.org/jira/browse/HBASE-25323) | Fix potential NPE when the zookeeper path of RegionServerTracker does not exist when start | Minor | . | +| [HBASE-25187](https://issues.apache.org/jira/browse/HBASE-25187) | Improve SizeCachedKV variants initialization | Minor | . | +| [HBASE-25261](https://issues.apache.org/jira/browse/HBASE-25261) | Upgrade Bootstrap to 3.4.1 | Major | security, UI | +| [HBASE-25267](https://issues.apache.org/jira/browse/HBASE-25267) | Add SSL keystore type and truststore related configs for HBase RESTServer | Major | REST | +| [HBASE-25003](https://issues.apache.org/jira/browse/HBASE-25003) | Backport HBASE-24350 and HBASE-24779 to branch-2.2 & branch-2.3 | Major | Replication | +| [HBASE-25240](https://issues.apache.org/jira/browse/HBASE-25240) | gson format of RpcServer.logResponse is abnormal | Minor | . | +| [HBASE-24859](https://issues.apache.org/jira/browse/HBASE-24859) | Optimize in-memory representation of mapreduce TableSplit objects | Major | mapreduce | +| [HBASE-25224](https://issues.apache.org/jira/browse/HBASE-25224) | Maximize sleep for checking meta and namespace regions availability | Major | master | +| [HBASE-25189](https://issues.apache.org/jira/browse/HBASE-25189) | [Metrics] Add checkAndPut and checkAndDelete latency metrics at table level | Minor | metrics | +| [HBASE-25179](https://issues.apache.org/jira/browse/HBASE-25179) | Assert format is incorrect in HFilePerformanceEvaluation class. | Minor | Performance, test | +| [HBASE-25146](https://issues.apache.org/jira/browse/HBASE-25146) | Add extra logging at info level to HFileCorruptionChecker
svn commit: r47113 [4/4] - /release/hbase/2.2.7/
Added: release/hbase/2.2.7/api_compare_2.2.7RC0_to_2.2.6.html
==
--- release/hbase/2.2.7/api_compare_2.2.7RC0_to_2.2.6.html (added)
+++ release/hbase/2.2.7/api_compare_2.2.7RC0_to_2.2.6.html Fri Apr 16 05:07:26
2021
@@ -0,0 +1,825 @@
+
+
+http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd";>
+http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en">
+
+
+
+
+
+hbase: rel/2.2.6 to 2.2.7RC0 compatibility report
+
+body {
+font-family:Arial, sans-serif;
+background-color:White;
+color:Black;
+}
+hr {
+color:Black;
+background-color:Black;
+height:1px;
+border:0;
+}
+h1 {
+margin-bottom:0px;
+padding-bottom:0px;
+font-size:1.625em;
+}
+h2 {
+margin-bottom:0px;
+padding-bottom:0px;
+font-size:1.25em;
+white-space:nowrap;
+}
+div.symbols {
+color:#003E69;
+}
+div.symbols i {
+color:Brown;
+}
+span.section {
+font-weight:bold;
+cursor:pointer;
+color:#003E69;
+white-space:nowrap;
+margin-left:0.3125em;
+}
+span:hover.section {
+color:#336699;
+}
+span.sect_aff {
+cursor:pointer;
+padding-left:1.55em;
+font-size:0.875em;
+color:#cc3300;
+}
+span.ext {
+font-weight:normal;
+}
+span.jar {
+color:#cc3300;
+font-size:0.875em;
+font-weight:bold;
+}
+div.jar_list {
+padding-left:0.4em;
+font-size:0.94em;
+}
+span.pkg_t {
+color:#408080;
+font-size:0.875em;
+}
+span.pkg {
+color:#408080;
+font-size:0.875em;
+font-weight:bold;
+}
+span.cname {
+color:Green;
+font-size:0.875em;
+font-weight:bold;
+}
+span.iname_b {
+font-weight:bold;
+}
+span.iname_a {
+color:#33;
+font-weight:bold;
+font-size:0.94em;
+}
+span.sym_p {
+font-weight:normal;
+white-space:normal;
+}
+span.sym_pd {
+white-space:normal;
+}
+span.sym_p span, span.sym_pd span {
+white-space:nowrap;
+}
+span.attr {
+color:Black;
+font-weight:normal;
+}
+span.deprecated {
+color:Red;
+font-weight:bold;
+font-family:Monaco, monospace;
+}
+div.affect {
+padding-left:1em;
+padding-bottom:10px;
+font-size:0.87em;
+font-style:italic;
+line-height:0.9em;
+}
+div.affected {
+padding-left:2em;
+padding-top:10px;
+}
+table.ptable {
+border-collapse:collapse;
+border:1px outset black;
+margin-left:0.95em;
+margin-top:3px;
+margin-bottom:3px;
+width:56.25em;
+}
+table.ptable td {
+border:1px solid Gray;
+padding:3px;
+font-size:0.875em;
+text-align:left;
+vertical-align:top;
+max-width:28em;
+word-wrap:break-word;
+}
+table.ptable th {
+background-color:#ee;
+font-weight:bold;
+color:#33;
+font-family:Verdana, Arial;
+font-size:0.875em;
+border:1px solid Gray;
+text-align:center;
+vertical-align:top;
+white-space:nowrap;
+padding:3px;
+}
+table.summary {
+border-collapse:collapse;
+border:1px outset black;
+}
+table.summary th {
+background-color:#ee;
+font-weight:normal;
+text-align:left;
+font-size:0.94em;
+white-space:nowrap;
+border:1px inset Gray;
+padding:3px;
+}
+table.summary td {
+text-align:right;
+white-space:nowrap;
+border:1px inset Gray;
+padding:3px 5px 3px 10px;
+}
+span.mngl {
+padding-left:1em;
+font-size:0.875em;
+cursor:text;
+color:#44;
+font-weight:bold;
+}
+span.pleft {
+padding-left:2.5em;
+}
+span.color_p {
+font-style:italic;
+color:Brown;
+}
+span.param {
+font-style:italic;
+}
+span.focus_p {
+font-style:italic;
+background-color:#DCDCDC;
+}
+span.ttype {
+font-weight:normal;
+}
+span.nowrap {
+white-space:nowrap;
+}
+span.value {
+white-space:nowrap;
+font-weight:bold;
+}
+.passed {
+background-color:#CCFFCC;
+font-weight:normal;
+}
+.warning {
+background-color:#F4F4AF;
+font-weight:normal;
+}
+.failed {
+background-color:#FF;
+font-weight:normal;
+}
+.new {
+background-color:#C6DEFF;
+font-weight:normal;
+}
+
+.compatible {
+background-color:#CCFFCC;
+font-weight:normal;
+}
+.almost_compatible {
+background-color:#FFDAA3;
+font-weight:normal;
+}
+.incompatible {
+background-color:#FF;
+font-weight:normal;
+}
+.gray {
+background-color:#DCDCDC;
+font-weight:normal;
+}
+
+.top_ref {
+font-size:0.69em;
+}
+.footer {
+font-size:0.8125em;
+}
+.tabset {
+float:left;
+}
+a.tab {
+border:1px solid Black;
+float:left;
+margin:0px 5px -1px 0px;
+padding:3px 5px 3px 5px;
+position:relative;
+font-size:0.875em;
+background-color:#DDD;
+text-decoration:none;
+color:Black;
+}
+a.disabled:hover
+{
+color:Black;
+background:#EEE;
+}
+a.active:hover
+{
+color:Black;
+background:White;
+}
+a.active {
+border-bottom-color:White;
+background-color:White;
+}
+div.tab {
+border-top:1px sol
svn commit: r47113 [3/4] - /release/hbase/2.2.7/
Added: release/hbase/2.2.7/RELEASENOTES.md == --- release/hbase/2.2.7/RELEASENOTES.md (added) +++ release/hbase/2.2.7/RELEASENOTES.md Fri Apr 16 05:07:26 2021 @@ -0,0 +1,2270 @@ +# RELEASENOTES + + +# HBASE 2.2.7 Release Notes + +These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements. + + +--- + +* [HBASE-25738](https://issues.apache.org/jira/browse/HBASE-25738) | *Minor* | **Backport HBASE-24305 to branch-2.2** + +The following method was added to ServerName + +- #valueOf(Address, long) + + +--- + +* [HBASE-25587](https://issues.apache.org/jira/browse/HBASE-25587) | *Major* | **[hbck2] Schedule SCP for all unknown servers** + +Adds scheduleSCPsForUnknownServers to Hbck Service. + + +--- + +* [HBASE-25460](https://issues.apache.org/jira/browse/HBASE-25460) | *Major* | **Expose drainingServers as cluster metric** + +Exposed new jmx metrics: "draininigRegionServers" and "numDrainingRegionServers" to provide "comma separated names for regionservers that are put in draining mode" and "num of such regionservers" respectively. + + +--- + +* [HBASE-25449](https://issues.apache.org/jira/browse/HBASE-25449) | *Major* | **'dfs.client.read.shortcircuit' should not be set in hbase-default.xml** + +The presence of HDFS short-circuit read configuration properties in hbase-default.xml inadvertently causes short-circuit reads to not happen inside of RegionServers, despite short-circuit reads being enabled in hdfs-site.xml. + + +--- + +* [HBASE-25441](https://issues.apache.org/jira/browse/HBASE-25441) | *Critical* | **add security check for some APIs in RSRpcServices** + +RsRpcServices APIs that can be accessed only through Admin rights: +- stopServer +- updateFavoredNodes +- updateConfiguration +- clearRegionBlockCache +- clearSlowLogsResponses + + +--- + +* [HBASE-25432](https://issues.apache.org/jira/browse/HBASE-25432) | *Blocker* | **we should add security checks for setTableStateInMeta and fixMeta** + +setTableStateInMeta and fixMeta can be accessed only through Admin rights + + +--- + +* [HBASE-25318](https://issues.apache.org/jira/browse/HBASE-25318) | *Minor* | **Configure where IntegrationTestImportTsv generates HFiles** + +Added IntegrationTestImportTsv.generatedHFileFolder configuration property to override the default location in IntegrationTestImportTsv. Useful for running the integration test when HDFS Transparent Encryption is enabled. + + +--- + +* [HBASE-25237](https://issues.apache.org/jira/browse/HBASE-25237) | *Major* | **'hbase master stop' shuts down the cluster, not the master only** + +\`hbase master stop\` should shutdown only master by default. +1. Help added to \`hbase master stop\`: +To stop cluster, use \`stop-hbase.sh\` or \`hbase master stop --shutDownCluster\` + +2. Help added to \`stop-hbase.sh\`: +stop-hbase.sh can only be used for shutting down entire cluster. To shut down (HMaster\|HRegionServer) use hbase-daemon.sh stop (master\|regionserver) + + +--- + +* [HBASE-25238](https://issues.apache.org/jira/browse/HBASE-25238) | *Critical* | **Upgrading HBase from 2.2.0 to 2.3.x fails because of âMessage missing required fields: stateâ** + +Fixes master procedure store migration issues going from 2.0.x to 2.2.x and/or 2.3.x. Also fixes failed heartbeat parse during rolling upgrade from 2.0.x. to 2.3.x. + + +--- + +* [HBASE-25224](https://issues.apache.org/jira/browse/HBASE-25224) | *Major* | **Maximize sleep for checking meta and namespace regions availability** + +Changed the max sleep time during meta and namespace regions availability check to be 60 sec. Previously there was no such cap + + +--- + +* [HBASE-25163](https://issues.apache.org/jira/browse/HBASE-25163) | *Major* | **Increase the timeout value for nightly jobs** + +Increase timeout value for nightly jobs to 16 hours since the new build machines are dedicated to hbase project, so we are allowed to use it all the time. + + +--- + +* [HBASE-22976](https://issues.apache.org/jira/browse/HBASE-22976) | *Major* | **[HBCK2] Add RecoveredEditsPlayer** + +WALPlayer can replay the content of recovered.edits directories. + +Side-effect is that WAL filename timestamp is now factored when setting start/end times for WALInputFormat; i.e. wal.start.time and wal.end.time values on a job context. Previous we looked at wal.end.time only. Now we consider wal.start.time too. If a file has a name outside of wal.start.time\<-\>wal.end.time, it'll be by-passed. This change-in-behavior will make it easier on operator crafting timestamp filters processing WALs. + + +--- + +* [HBASE-25154](https://issues.apache.org/jira/browse/HBASE-25154) | *Major* | **Set java.io.tmpdir to project build directory to avoid writing std\*deferred files to /tmp** + +Change the java.io.tmpdir to project.build.directory in surefire-maven-plugin, to avoid writing std\*deferred files to /tmp w
svn commit: r46981 [3/4] - /dev/hbase/2.2.7RC0/
Added: dev/hbase/2.2.7RC0/RELEASENOTES.md == --- dev/hbase/2.2.7RC0/RELEASENOTES.md (added) +++ dev/hbase/2.2.7RC0/RELEASENOTES.md Sun Apr 11 11:59:34 2021 @@ -0,0 +1,2270 @@ +# RELEASENOTES + + +# HBASE 2.2.7 Release Notes + +These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements. + + +--- + +* [HBASE-25738](https://issues.apache.org/jira/browse/HBASE-25738) | *Minor* | **Backport HBASE-24305 to branch-2.2** + +The following method was added to ServerName + +- #valueOf(Address, long) + + +--- + +* [HBASE-25587](https://issues.apache.org/jira/browse/HBASE-25587) | *Major* | **[hbck2] Schedule SCP for all unknown servers** + +Adds scheduleSCPsForUnknownServers to Hbck Service. + + +--- + +* [HBASE-25460](https://issues.apache.org/jira/browse/HBASE-25460) | *Major* | **Expose drainingServers as cluster metric** + +Exposed new jmx metrics: "draininigRegionServers" and "numDrainingRegionServers" to provide "comma separated names for regionservers that are put in draining mode" and "num of such regionservers" respectively. + + +--- + +* [HBASE-25449](https://issues.apache.org/jira/browse/HBASE-25449) | *Major* | **'dfs.client.read.shortcircuit' should not be set in hbase-default.xml** + +The presence of HDFS short-circuit read configuration properties in hbase-default.xml inadvertently causes short-circuit reads to not happen inside of RegionServers, despite short-circuit reads being enabled in hdfs-site.xml. + + +--- + +* [HBASE-25441](https://issues.apache.org/jira/browse/HBASE-25441) | *Critical* | **add security check for some APIs in RSRpcServices** + +RsRpcServices APIs that can be accessed only through Admin rights: +- stopServer +- updateFavoredNodes +- updateConfiguration +- clearRegionBlockCache +- clearSlowLogsResponses + + +--- + +* [HBASE-25432](https://issues.apache.org/jira/browse/HBASE-25432) | *Blocker* | **we should add security checks for setTableStateInMeta and fixMeta** + +setTableStateInMeta and fixMeta can be accessed only through Admin rights + + +--- + +* [HBASE-25318](https://issues.apache.org/jira/browse/HBASE-25318) | *Minor* | **Configure where IntegrationTestImportTsv generates HFiles** + +Added IntegrationTestImportTsv.generatedHFileFolder configuration property to override the default location in IntegrationTestImportTsv. Useful for running the integration test when HDFS Transparent Encryption is enabled. + + +--- + +* [HBASE-25237](https://issues.apache.org/jira/browse/HBASE-25237) | *Major* | **'hbase master stop' shuts down the cluster, not the master only** + +\`hbase master stop\` should shutdown only master by default. +1. Help added to \`hbase master stop\`: +To stop cluster, use \`stop-hbase.sh\` or \`hbase master stop --shutDownCluster\` + +2. Help added to \`stop-hbase.sh\`: +stop-hbase.sh can only be used for shutting down entire cluster. To shut down (HMaster\|HRegionServer) use hbase-daemon.sh stop (master\|regionserver) + + +--- + +* [HBASE-25238](https://issues.apache.org/jira/browse/HBASE-25238) | *Critical* | **Upgrading HBase from 2.2.0 to 2.3.x fails because of âMessage missing required fields: stateâ** + +Fixes master procedure store migration issues going from 2.0.x to 2.2.x and/or 2.3.x. Also fixes failed heartbeat parse during rolling upgrade from 2.0.x. to 2.3.x. + + +--- + +* [HBASE-25224](https://issues.apache.org/jira/browse/HBASE-25224) | *Major* | **Maximize sleep for checking meta and namespace regions availability** + +Changed the max sleep time during meta and namespace regions availability check to be 60 sec. Previously there was no such cap + + +--- + +* [HBASE-25163](https://issues.apache.org/jira/browse/HBASE-25163) | *Major* | **Increase the timeout value for nightly jobs** + +Increase timeout value for nightly jobs to 16 hours since the new build machines are dedicated to hbase project, so we are allowed to use it all the time. + + +--- + +* [HBASE-22976](https://issues.apache.org/jira/browse/HBASE-22976) | *Major* | **[HBCK2] Add RecoveredEditsPlayer** + +WALPlayer can replay the content of recovered.edits directories. + +Side-effect is that WAL filename timestamp is now factored when setting start/end times for WALInputFormat; i.e. wal.start.time and wal.end.time values on a job context. Previous we looked at wal.end.time only. Now we consider wal.start.time too. If a file has a name outside of wal.start.time\<-\>wal.end.time, it'll be by-passed. This change-in-behavior will make it easier on operator crafting timestamp filters processing WALs. + + +--- + +* [HBASE-25154](https://issues.apache.org/jira/browse/HBASE-25154) | *Major* | **Set java.io.tmpdir to project build directory to avoid writing std\*deferred files to /tmp** + +Change the java.io.tmpdir to project.build.directory in surefire-maven-plugin, to avoid writing std\*deferred files to /tmp whic
svn commit: r46981 [4/4] - /dev/hbase/2.2.7RC0/
Added: dev/hbase/2.2.7RC0/api_compare_2.2.7RC0_to_2.2.6.html
==
--- dev/hbase/2.2.7RC0/api_compare_2.2.7RC0_to_2.2.6.html (added)
+++ dev/hbase/2.2.7RC0/api_compare_2.2.7RC0_to_2.2.6.html Sun Apr 11 11:59:34
2021
@@ -0,0 +1,825 @@
+
+
+http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd";>
+http://www.w3.org/1999/xhtml"; xml:lang="en" lang="en">
+
+
+
+
+
+hbase: rel/2.2.6 to 2.2.7RC0 compatibility report
+
+body {
+font-family:Arial, sans-serif;
+background-color:White;
+color:Black;
+}
+hr {
+color:Black;
+background-color:Black;
+height:1px;
+border:0;
+}
+h1 {
+margin-bottom:0px;
+padding-bottom:0px;
+font-size:1.625em;
+}
+h2 {
+margin-bottom:0px;
+padding-bottom:0px;
+font-size:1.25em;
+white-space:nowrap;
+}
+div.symbols {
+color:#003E69;
+}
+div.symbols i {
+color:Brown;
+}
+span.section {
+font-weight:bold;
+cursor:pointer;
+color:#003E69;
+white-space:nowrap;
+margin-left:0.3125em;
+}
+span:hover.section {
+color:#336699;
+}
+span.sect_aff {
+cursor:pointer;
+padding-left:1.55em;
+font-size:0.875em;
+color:#cc3300;
+}
+span.ext {
+font-weight:normal;
+}
+span.jar {
+color:#cc3300;
+font-size:0.875em;
+font-weight:bold;
+}
+div.jar_list {
+padding-left:0.4em;
+font-size:0.94em;
+}
+span.pkg_t {
+color:#408080;
+font-size:0.875em;
+}
+span.pkg {
+color:#408080;
+font-size:0.875em;
+font-weight:bold;
+}
+span.cname {
+color:Green;
+font-size:0.875em;
+font-weight:bold;
+}
+span.iname_b {
+font-weight:bold;
+}
+span.iname_a {
+color:#33;
+font-weight:bold;
+font-size:0.94em;
+}
+span.sym_p {
+font-weight:normal;
+white-space:normal;
+}
+span.sym_pd {
+white-space:normal;
+}
+span.sym_p span, span.sym_pd span {
+white-space:nowrap;
+}
+span.attr {
+color:Black;
+font-weight:normal;
+}
+span.deprecated {
+color:Red;
+font-weight:bold;
+font-family:Monaco, monospace;
+}
+div.affect {
+padding-left:1em;
+padding-bottom:10px;
+font-size:0.87em;
+font-style:italic;
+line-height:0.9em;
+}
+div.affected {
+padding-left:2em;
+padding-top:10px;
+}
+table.ptable {
+border-collapse:collapse;
+border:1px outset black;
+margin-left:0.95em;
+margin-top:3px;
+margin-bottom:3px;
+width:56.25em;
+}
+table.ptable td {
+border:1px solid Gray;
+padding:3px;
+font-size:0.875em;
+text-align:left;
+vertical-align:top;
+max-width:28em;
+word-wrap:break-word;
+}
+table.ptable th {
+background-color:#ee;
+font-weight:bold;
+color:#33;
+font-family:Verdana, Arial;
+font-size:0.875em;
+border:1px solid Gray;
+text-align:center;
+vertical-align:top;
+white-space:nowrap;
+padding:3px;
+}
+table.summary {
+border-collapse:collapse;
+border:1px outset black;
+}
+table.summary th {
+background-color:#ee;
+font-weight:normal;
+text-align:left;
+font-size:0.94em;
+white-space:nowrap;
+border:1px inset Gray;
+padding:3px;
+}
+table.summary td {
+text-align:right;
+white-space:nowrap;
+border:1px inset Gray;
+padding:3px 5px 3px 10px;
+}
+span.mngl {
+padding-left:1em;
+font-size:0.875em;
+cursor:text;
+color:#44;
+font-weight:bold;
+}
+span.pleft {
+padding-left:2.5em;
+}
+span.color_p {
+font-style:italic;
+color:Brown;
+}
+span.param {
+font-style:italic;
+}
+span.focus_p {
+font-style:italic;
+background-color:#DCDCDC;
+}
+span.ttype {
+font-weight:normal;
+}
+span.nowrap {
+white-space:nowrap;
+}
+span.value {
+white-space:nowrap;
+font-weight:bold;
+}
+.passed {
+background-color:#CCFFCC;
+font-weight:normal;
+}
+.warning {
+background-color:#F4F4AF;
+font-weight:normal;
+}
+.failed {
+background-color:#FF;
+font-weight:normal;
+}
+.new {
+background-color:#C6DEFF;
+font-weight:normal;
+}
+
+.compatible {
+background-color:#CCFFCC;
+font-weight:normal;
+}
+.almost_compatible {
+background-color:#FFDAA3;
+font-weight:normal;
+}
+.incompatible {
+background-color:#FF;
+font-weight:normal;
+}
+.gray {
+background-color:#DCDCDC;
+font-weight:normal;
+}
+
+.top_ref {
+font-size:0.69em;
+}
+.footer {
+font-size:0.8125em;
+}
+.tabset {
+float:left;
+}
+a.tab {
+border:1px solid Black;
+float:left;
+margin:0px 5px -1px 0px;
+padding:3px 5px 3px 5px;
+position:relative;
+font-size:0.875em;
+background-color:#DDD;
+text-decoration:none;
+color:Black;
+}
+a.disabled:hover
+{
+color:Black;
+background:#EEE;
+}
+a.active:hover
+{
+color:Black;
+background:White;
+}
+a.active {
+border-bottom-color:White;
+background-color:White;
+}
+div.tab {
+border-top:1px solid
svn commit: r46981 [2/4] - /dev/hbase/2.2.7RC0/
Added: dev/hbase/2.2.7RC0/CHANGES.md == --- dev/hbase/2.2.7RC0/CHANGES.md (added) +++ dev/hbase/2.2.7RC0/CHANGES.md Sun Apr 11 11:59:34 2021 @@ -0,0 +1,1721 @@ +# HBASE Changelog + + +## Release 2.2.7 - Unreleased (as of 2021-04-11) + + + +### NEW FEATURES: + +| JIRA | Summary | Priority | Component | +|: |: | :--- |: | +| [HBASE-25587](https://issues.apache.org/jira/browse/HBASE-25587) | [hbck2] Schedule SCP for all unknown servers | Major | hbase-operator-tools, hbck2 | +| [HBASE-25460](https://issues.apache.org/jira/browse/HBASE-25460) | Expose drainingServers as cluster metric | Major | metrics | +| [HBASE-24776](https://issues.apache.org/jira/browse/HBASE-24776) | [hbtop] Support Batch mode | Major | hbtop | + + +### IMPROVEMENTS: + +| JIRA | Summary | Priority | Component | +|: |: | :--- |: | +| [HBASE-25656](https://issues.apache.org/jira/browse/HBASE-25656) | Backport to branch-2.2: [HBASE-25548 Optionally allow snapshots to preserve cluster's max filesize config by setting it into table descriptor] | Major | . | +| [HBASE-25621](https://issues.apache.org/jira/browse/HBASE-25621) | Balancer should check region plan source to avoid misplace region groups | Major | Balancer | +| [HBASE-25541](https://issues.apache.org/jira/browse/HBASE-25541) | In WALEntryStream, set the current path to null while dequeing the log | Major | . | +| [HBASE-25536](https://issues.apache.org/jira/browse/HBASE-25536) | Remove 0 length wal file from logQueue if it belongs to old sources. | Major | Replication | +| [HBASE-25329](https://issues.apache.org/jira/browse/HBASE-25329) | Dump region hashes in logs for the regions that are stuck in transition for more than a configured amount of time | Minor | . | +| [HBASE-25475](https://issues.apache.org/jira/browse/HBASE-25475) | Improve unit test for HBASE-25445 : SplitWALRemoteProcedure failed to archive split WAL | Minor | wal | +| [HBASE-25449](https://issues.apache.org/jira/browse/HBASE-25449) | 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml | Major | conf | +| [HBASE-25483](https://issues.apache.org/jira/browse/HBASE-25483) | set the loadMeta log level to debug. | Major | MTTR, Region Assignment | +| [HBASE-25318](https://issues.apache.org/jira/browse/HBASE-25318) | Configure where IntegrationTestImportTsv generates HFiles | Minor | integration tests | +| [HBASE-24850](https://issues.apache.org/jira/browse/HBASE-24850) | CellComparator perf improvement | Critical | Performance, scan | +| [HBASE-25425](https://issues.apache.org/jira/browse/HBASE-25425) | Some notes on RawCell | Trivial | . | +| [HBASE-25246](https://issues.apache.org/jira/browse/HBASE-25246) | Backup/Restore hbase cell tags. | Major | backup&restore | +| [HBASE-25328](https://issues.apache.org/jira/browse/HBASE-25328) | Add builder method to create Tags. | Minor | . | +| [HBASE-25339](https://issues.apache.org/jira/browse/HBASE-25339) | Method parameter and member variable are duplicated in checkSplittable() of SplitTableRegionProcedure | Minor | . | +| [HBASE-25237](https://issues.apache.org/jira/browse/HBASE-25237) | 'hbase master stop' shuts down the cluster, not the master only | Major | . | +| [HBASE-25323](https://issues.apache.org/jira/browse/HBASE-25323) | Fix potential NPE when the zookeeper path of RegionServerTracker does not exist when start | Minor | . | +| [HBASE-25187](https://issues.apache.org/jira/browse/HBASE-25187) | Improve SizeCachedKV variants initialization | Minor | . | +| [HBASE-25261](https://issues.apache.org/jira/browse/HBASE-25261) | Upgrade Bootstrap to 3.4.1 | Major | security, UI | +| [HBASE-25267](https://issues.apache.org/jira/browse/HBASE-25267) | Add SSL keystore type and truststore related configs for HBase RESTServer | Major | REST | +| [HBASE-25003](https://issues.apache.org/jira/browse/HBASE-25003) | Backport HBASE-24350 and HBASE-24779 to branch-2.2 & branch-2.3 | Major | Replication | +| [HBASE-25240](https://issues.apache.org/jira/browse/HBASE-25240) | gson format of RpcServer.logResponse is abnormal | Minor | . | +| [HBASE-24859](https://issues.apache.org/jira/browse/HBASE-24859) | Optimize in-memory representation of mapreduce TableSplit objects | Major | mapreduce | +| [HBASE-25224](https://issues.apache.org/jira/browse/HBASE-25224) | Maximize sleep for checking meta and namespace regions availability | Major | master | +| [HBASE-25189](https://issues.apache.org/jira/browse/HBASE-25189) | [Metrics] Add checkAndPut and checkAndDelete latency metrics at table level | Minor | metrics | +| [HBASE-25179](https://issues.apache.org/jira/browse/HBASE-25179) | Assert format is incorrect in HFilePerformanceEvaluation class. | Minor | Performance, test | +| [HBASE-25146](https://issues.apache.org/jira/browse/HBASE-25146) | Add extra logging at info level to HFileCorruptionChecker in
svn commit: r46981 [1/4] - /dev/hbase/2.2.7RC0/
Author: zghao Date: Sun Apr 11 11:59:34 2021 New Revision: 46981 Log: Apache HBase 2.2.7RC0 Added: dev/hbase/2.2.7RC0/ dev/hbase/2.2.7RC0/CHANGES.md dev/hbase/2.2.7RC0/RELEASENOTES.md dev/hbase/2.2.7RC0/api_compare_2.2.7RC0_to_2.2.6.html dev/hbase/2.2.7RC0/hbase-2.2.7-bin.tar.gz (with props) dev/hbase/2.2.7RC0/hbase-2.2.7-bin.tar.gz.asc dev/hbase/2.2.7RC0/hbase-2.2.7-bin.tar.gz.sha512 dev/hbase/2.2.7RC0/hbase-2.2.7-client-bin.tar.gz (with props) dev/hbase/2.2.7RC0/hbase-2.2.7-client-bin.tar.gz.asc dev/hbase/2.2.7RC0/hbase-2.2.7-client-bin.tar.gz.sha512 dev/hbase/2.2.7RC0/hbase-2.2.7-src.tar.gz (with props) dev/hbase/2.2.7RC0/hbase-2.2.7-src.tar.gz.asc dev/hbase/2.2.7RC0/hbase-2.2.7-src.tar.gz.sha512
[hbase] annotated tag 2.2.7RC0 created (now ad1f8a0)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to annotated tag 2.2.7RC0 in repository https://gitbox.apache.org/repos/asf/hbase.git. at ad1f8a0 (tag) tagging 0fc18a9056e5eb3a80fdbde916865607946c5195 (commit) replaces rel/2.2.6 by Guanghao Zhang on Sun Apr 11 19:22:36 2021 +0800 - Log - Tagging the 2.2.7 first Release Candidate -BEGIN PGP SIGNATURE- iQIzBAABCAAdFiEEqgCu7gDR/ij/88WIApVEXRfxNG4FAmBy2/wACgkQApVEXRfx NG6AkA//cYoO2p9EwAaNYc7craQaomiyDZMZSkJ235kdDy1rfBeHjZ1s3e4NQ+il a4gj30uNYc4z46OElP+HFDc1tzcnhD1QwEnPu1obiBP8O3VHpe68azv1fT5EYdNc nYAYwUsK2ZvWm2VieDD1vMKiu27k5nMjWjTd/m8T/uJUivcjut15HrIgHNlvsngH x2lEfV10rvfk1jZcQYziDQSAkryxtQQAYEoqk6bJmm7JjTHYW15h/R4QZYr9QtD+ ts1VQ+CN3mJ8DlEya3a8FP+iYHcxviPltGXQT0YqEpl4oZnpgy2UalZBPcesRvI+ LjEGevte6bmg1HODSL4SFK69PVbGXDhupTvYGYX0FrHYXduCYv+XCivggRO0WrWY yN7tiluloWgWKNIPdgPpUX9YX/fwvTIa1zxAvUDKGgDuFmvYrhRC+y4UPXYtEqpS ffQkBnN+i3ciKl50VK5IkUSKBZJT0IORz8xVTDZVLsr9oaTDy0cqWp8tNWAHARkF J0GcU3zB36MGGxq/mOaBXlqsNBgUJ3xhHHeRVSPobkHoEgm3wUASyOkURMEpHQPL rW0WQV4RrZbJpYVxldhHa7JIfmMGVZh5KKXIPNHrrROi/h9Am79c1au5NmPyd0Wd SqNpvypojo/JEVPOiBKXQGBjz/tyOpGvRz67vEKi+bQ0kCMEMKU= =I/GE -END PGP SIGNATURE- --- No new revisions were added by this update.
[hbase] branch branch-2.2 updated: HBASE-25765 Set version as 2.2.7 in branch-2.2 (#3148)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a commit to branch branch-2.2 in repository https://gitbox.apache.org/repos/asf/hbase.git The following commit(s) were added to refs/heads/branch-2.2 by this push: new 0fc18a9 HBASE-25765 Set version as 2.2.7 in branch-2.2 (#3148) 0fc18a9 is described below commit 0fc18a9056e5eb3a80fdbde916865607946c5195 Author: Guanghao Zhang AuthorDate: Sun Apr 11 19:15:07 2021 +0800 HBASE-25765 Set version as 2.2.7 in branch-2.2 (#3148) --- hbase-annotations/pom.xml | 2 +- hbase-archetypes/hbase-archetype-builder/pom.xml | 2 +- hbase-archetypes/hbase-client-project/pom.xml | 2 +- hbase-archetypes/hbase-shaded-client-project/pom.xml | 2 +- hbase-archetypes/pom.xml | 2 +- hbase-assembly/pom.xml | 2 +- hbase-build-configuration/pom.xml | 2 +- hbase-checkstyle/pom.xml | 4 ++-- hbase-client/pom.xml | 2 +- hbase-common/pom.xml | 2 +- hbase-endpoint/pom.xml | 2 +- hbase-examples/pom.xml | 2 +- hbase-external-blockcache/pom.xml | 2 +- hbase-hadoop-compat/pom.xml| 2 +- hbase-hadoop2-compat/pom.xml | 2 +- hbase-hbtop/pom.xml| 2 +- hbase-http/pom.xml | 2 +- hbase-it/pom.xml | 2 +- hbase-mapreduce/pom.xml| 2 +- hbase-metrics-api/pom.xml | 2 +- hbase-metrics/pom.xml | 2 +- hbase-procedure/pom.xml| 2 +- hbase-protocol-shaded/pom.xml | 2 +- hbase-protocol/pom.xml | 2 +- hbase-replication/pom.xml | 2 +- hbase-resource-bundle/pom.xml | 2 +- hbase-rest/pom.xml | 2 +- hbase-rsgroup/pom.xml | 2 +- hbase-server/pom.xml | 2 +- hbase-shaded/hbase-shaded-check-invariants/pom.xml | 2 +- hbase-shaded/hbase-shaded-client-byo-hadoop/pom.xml| 2 +- hbase-shaded/hbase-shaded-client/pom.xml | 2 +- hbase-shaded/hbase-shaded-mapreduce/pom.xml| 2 +- hbase-shaded/hbase-shaded-testing-util-tester/pom.xml | 2 +- hbase-shaded/hbase-shaded-testing-util/pom.xml | 2 +- hbase-shaded/hbase-shaded-with-hadoop-check-invariants/pom.xml | 2 +- hbase-shaded/pom.xml | 2 +- hbase-shell/pom.xml| 2 +- hbase-testing-util/pom.xml | 2 +- hbase-thrift/pom.xml | 2 +- hbase-zookeeper/pom.xml| 2 +- pom.xml| 2 +- 42 files changed, 43 insertions(+), 43 deletions(-) diff --git a/hbase-annotations/pom.xml b/hbase-annotations/pom.xml index 08219df..2dd5403 100644 --- a/hbase-annotations/pom.xml +++ b/hbase-annotations/pom.xml @@ -23,7 +23,7 @@ hbase org.apache.hbase -2.2.7-SNAPSHOT +2.2.7 .. diff --git a/hbase-archetypes/hbase-archetype-builder/pom.xml b/hbase-archetypes/hbase-archetype-builder/pom.xml index 4ff0850..be0cf80 100644 --- a/hbase-archetypes/hbase-archetype-builder/pom.xml +++ b/hbase-archetypes/hbase-archetype-builder/pom.xml @@ -25,7 +25,7 @@ hbase-archetypes org.apache.hbase -2.2.7-SNAPSHOT +2.2.7 .. diff --git a/hbase-archetypes/hbase-client-project/pom.xml b/hbase-archetypes/hbase-client-project/pom.xml index 30c9e57..c98ea0d 100644 --- a/hbase-archetypes/hbase-client-project/pom.xml +++ b/hbase-archetypes/hbase-client-project/pom.xml @@ -26,7 +26,7 @@ hbase-archetypes org.apache.hbase -2.2.7-SNAPSHOT +2.2.7 .. hbase-client-project diff --git a/hbase-archetypes/hbase-shaded-client-project/pom.xml b/hbase-archetypes/hbase-shaded-client-project/pom.xml index 01a70fd..a948375 100644 --- a/hbase-archetypes/hbase-shaded-client-project/pom.xml +++ b/hbase-archetypes/hbase-shaded-client-project/pom.xml @@ -26,7 +26,7 @@ hbase-archetypes org.apache.hbase -2.2.7-SNAPSHOT +2.2.7 .. hbase-shaded-client-project diff --git a/hbase
[hbase] branch branch-2.2 updated: HBASE-25764 Generate CHANGES.md and RELEASENOTES.md for 2.2.7 (#3147)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a commit to branch branch-2.2 in repository https://gitbox.apache.org/repos/asf/hbase.git The following commit(s) were added to refs/heads/branch-2.2 by this push: new b6b3c80 HBASE-25764 Generate CHANGES.md and RELEASENOTES.md for 2.2.7 (#3147) b6b3c80 is described below commit b6b3c80e4f8c52d14a735be2c6466b5992b0563b Author: Guanghao Zhang AuthorDate: Sun Apr 11 18:11:54 2021 +0800 HBASE-25764 Generate CHANGES.md and RELEASENOTES.md for 2.2.7 (#3147) --- CHANGES.md | 179 RELEASENOTES.md | 158 + 2 files changed, 337 insertions(+) diff --git a/CHANGES.md b/CHANGES.md index e754cc1..7e332a1 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -20,6 +20,185 @@ # Be careful doing manual edits in this file. Do not change format # of release header or remove the below marker. This file is generated. # DO NOT REMOVE THIS MARKER; FOR INTERPOLATING CHANGES!--> +## Release 2.2.7 - Unreleased (as of 2021-04-11) + + + +### NEW FEATURES: + +| JIRA | Summary | Priority | Component | +|: |: | :--- |: | +| [HBASE-25587](https://issues.apache.org/jira/browse/HBASE-25587) | [hbck2] Schedule SCP for all unknown servers | Major | hbase-operator-tools, hbck2 | +| [HBASE-25460](https://issues.apache.org/jira/browse/HBASE-25460) | Expose drainingServers as cluster metric | Major | metrics | +| [HBASE-24776](https://issues.apache.org/jira/browse/HBASE-24776) | [hbtop] Support Batch mode | Major | hbtop | + + +### IMPROVEMENTS: + +| JIRA | Summary | Priority | Component | +|: |: | :--- |: | +| [HBASE-25656](https://issues.apache.org/jira/browse/HBASE-25656) | Backport to branch-2.2: [HBASE-25548 Optionally allow snapshots to preserve cluster's max filesize config by setting it into table descriptor] | Major | . | +| [HBASE-25621](https://issues.apache.org/jira/browse/HBASE-25621) | Balancer should check region plan source to avoid misplace region groups | Major | Balancer | +| [HBASE-25541](https://issues.apache.org/jira/browse/HBASE-25541) | In WALEntryStream, set the current path to null while dequeing the log | Major | . | +| [HBASE-25536](https://issues.apache.org/jira/browse/HBASE-25536) | Remove 0 length wal file from logQueue if it belongs to old sources. | Major | Replication | +| [HBASE-25329](https://issues.apache.org/jira/browse/HBASE-25329) | Dump region hashes in logs for the regions that are stuck in transition for more than a configured amount of time | Minor | . | +| [HBASE-25475](https://issues.apache.org/jira/browse/HBASE-25475) | Improve unit test for HBASE-25445 : SplitWALRemoteProcedure failed to archive split WAL | Minor | wal | +| [HBASE-25449](https://issues.apache.org/jira/browse/HBASE-25449) | 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml | Major | conf | +| [HBASE-25483](https://issues.apache.org/jira/browse/HBASE-25483) | set the loadMeta log level to debug. | Major | MTTR, Region Assignment | +| [HBASE-25318](https://issues.apache.org/jira/browse/HBASE-25318) | Configure where IntegrationTestImportTsv generates HFiles | Minor | integration tests | +| [HBASE-24850](https://issues.apache.org/jira/browse/HBASE-24850) | CellComparator perf improvement | Critical | Performance, scan | +| [HBASE-25425](https://issues.apache.org/jira/browse/HBASE-25425) | Some notes on RawCell | Trivial | . | +| [HBASE-25246](https://issues.apache.org/jira/browse/HBASE-25246) | Backup/Restore hbase cell tags. | Major | backup&restore | +| [HBASE-25328](https://issues.apache.org/jira/browse/HBASE-25328) | Add builder method to create Tags. | Minor | . | +| [HBASE-25339](https://issues.apache.org/jira/browse/HBASE-25339) | Method parameter and member variable are duplicated in checkSplittable() of SplitTableRegionProcedure | Minor | . | +| [HBASE-25237](https://issues.apache.org/jira/browse/HBASE-25237) | 'hbase master stop' shuts down the cluster, not the master only | Major | . | +| [HBASE-25323](https://issues.apache.org/jira/browse/HBASE-25323) | Fix potential NPE when the zookeeper path of RegionServerTracker does not exist when start | Minor | . | +| [HBASE-25187](https://issues.apache.org/jira/browse/HBASE-25187) | Improve SizeCachedKV variants initialization | Minor | . | +| [HBASE-25261](https://issues.apache.org/jira/browse/HBASE-25261) | Upgrade Bootstrap to 3.4.1 | Major | security, UI | +| [HBASE-25267](https://issues.apache.org/jira/browse/HBASE-25267) | Add SSL keystore type and truststore related configs for HBase RESTServer | Major | REST | +| [HBASE-25003](https://issues.apache.org/jira/browse/HBASE-25003) | Backport HBASE-24350 and HBASE-24779 to branch-2.2 & branch-2.3 | Major | Replication | +| [HBASE-25240](https://issues.apache.org/jira/browse/HBASE-25240) | gson
[hbase] branch branch-2.2 updated: HBASE-25605 Try ignore the ExportSnapshot related unit tests for branch-2.2 (#2985)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch branch-2.2
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-2.2 by this push:
new b08c133 HBASE-25605 Try ignore the ExportSnapshot related unit tests
for branch-2.2 (#2985)
b08c133 is described below
commit b08c13364b476414f4da6e274813d869d38680ef
Author: Guanghao Zhang
AuthorDate: Fri Feb 26 10:40:32 2021 +0800
HBASE-25605 Try ignore the ExportSnapshot related unit tests for branch-2.2
(#2985)
Signed-off-by: Duo Zhang
---
.../test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java | 2 ++
.../org/apache/hadoop/hbase/snapshot/TestExportSnapshotNoCluster.java | 2 ++
.../hadoop/hbase/snapshot/TestExportSnapshotWithTemporaryDirectory.java | 2 ++
.../java/org/apache/hadoop/hbase/snapshot/TestMobExportSnapshot.java| 2 ++
.../org/apache/hadoop/hbase/snapshot/TestMobSecureExportSnapshot.java | 2 ++
.../java/org/apache/hadoop/hbase/snapshot/TestSecureExportSnapshot.java | 2 ++
.../java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java | 2 +-
7 files changed, 13 insertions(+), 1 deletion(-)
diff --git
a/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
b/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
index c988854..9b1320d 100644
---
a/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
+++
b/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
@@ -49,6 +49,7 @@ import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.ClassRule;
+import org.junit.Ignore;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
@@ -62,6 +63,7 @@ import
org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.Snapshot
/**
* Test Export Snapshot Tool
*/
+@Ignore
@Category({VerySlowMapReduceTests.class, LargeTests.class})
public class TestExportSnapshot {
diff --git
a/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotNoCluster.java
b/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotNoCluster.java
index f295e90..1c7ddf8 100644
---
a/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotNoCluster.java
+++
b/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotNoCluster.java
@@ -34,6 +34,7 @@ import org.apache.hadoop.hbase.testclassification.MediumTests;
import org.apache.hadoop.hbase.util.Bytes;
import org.junit.Before;
import org.junit.ClassRule;
+import org.junit.Ignore;
import org.junit.Test;
import org.junit.experimental.categories.Category;
import org.slf4j.Logger;
@@ -42,6 +43,7 @@ import org.slf4j.LoggerFactory;
/**
* Test Export Snapshot Tool
*/
+@Ignore
@Category({MapReduceTests.class, MediumTests.class})
public class TestExportSnapshotNoCluster {
diff --git
a/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotWithTemporaryDirectory.java
b/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotWithTemporaryDirectory.java
index fca5358..d7a4ff2 100644
---
a/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotWithTemporaryDirectory.java
+++
b/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotWithTemporaryDirectory.java
@@ -28,8 +28,10 @@ import
org.apache.hadoop.hbase.testclassification.MediumTests;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.ClassRule;
+import org.junit.Ignore;
import org.junit.experimental.categories.Category;
+@Ignore
@Category({MediumTests.class})
public class TestExportSnapshotWithTemporaryDirectory extends
TestExportSnapshot {
diff --git
a/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestMobExportSnapshot.java
b/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestMobExportSnapshot.java
index 59cdf4d..38ef1e8 100644
---
a/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestMobExportSnapshot.java
+++
b/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestMobExportSnapshot.java
@@ -26,11 +26,13 @@ import
org.apache.hadoop.hbase.testclassification.LargeTests;
import org.apache.hadoop.hbase.testclassification.VerySlowRegionServerTests;
import org.junit.BeforeClass;
import org.junit.ClassRule;
+import org.junit.Ignore;
import org.junit.experimental.categories.Category;
/**
* Test Export Snapshot Tool
*/
+@Ignore
@Category({VerySlowRegionServerTests.class, LargeTests.class})
public class TestMobExportSnapshot extends TestExportSnapshot {
diff --git
a/hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestMobSecureExportSnapshot.java
b
[hbase] branch branch-2.2 updated: HBASE-25545 Fix the build problem for branch-2.2 (#2920)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch branch-2.2
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-2.2 by this push:
new 52c9270 HBASE-25545 Fix the build problem for branch-2.2 (#2920)
52c9270 is described below
commit 52c9270639458bc9dad1466d2d5245c8b8f722d2
Author: Guanghao Zhang
AuthorDate: Wed Feb 3 10:37:28 2021 +0800
HBASE-25545 Fix the build problem for branch-2.2 (#2920)
Signed-off-by: meiyi
---
.../apache/hadoop/hbase/master/TestRetainAssignmentOnRestart.java | 6 +-
1 file changed, 1 insertion(+), 5 deletions(-)
diff --git
a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRetainAssignmentOnRestart.java
b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRetainAssignmentOnRestart.java
index ee4da74..3881897 100644
---
a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRetainAssignmentOnRestart.java
+++
b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRetainAssignmentOnRestart.java
@@ -199,10 +199,6 @@ public class TestRetainAssignmentOnRestart extends
AbstractTestRestartCluster {
}
private void setupCluster() throws Exception, IOException,
InterruptedException {
-// Set Zookeeper based connection registry since we will stop master and
start a new master
-// without populating the underlying config for the connection.
-
UTIL.getConfiguration().set(HConstants.CLIENT_CONNECTION_REGISTRY_IMPL_CONF_KEY,
- HConstants.ZK_CONNECTION_REGISTRY_CLASS);
// Enable retain assignment during ServerCrashProcedure
UTIL.getConfiguration().setBoolean(ServerCrashProcedure.MASTER_SCP_RETAIN_ASSIGNMENT,
true);
UTIL.startMiniCluster(NUM_OF_RS);
@@ -237,4 +233,4 @@ public class TestRetainAssignmentOnRestart extends
AbstractTestRestartCluster {
assertTrue(found);
}
}
-}
\ No newline at end of file
+}
[hbase] branch branch-2.2 updated: HBASE-25277 postScannerFilterRow impacts Scan performance a lot in HBase 2.x (#2765)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch branch-2.2
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-2.2 by this push:
new 4cd6047 HBASE-25277 postScannerFilterRow impacts Scan performance a
lot in HBase 2.x (#2765)
4cd6047 is described below
commit 4cd60474e1dcaa1856838abd1f677c99baa59284
Author: Pankaj
AuthorDate: Tue Dec 22 06:20:33 2020 +0530
HBASE-25277 postScannerFilterRow impacts Scan performance a lot in HBase
2.x (#2765)
Signed-off-by: Guanghao Zhang
---
.../hadoop/hbase/constraint/ConstraintProcessor.java | 18 +-
.../hbase/regionserver/RegionCoprocessorHost.java | 16 +++-
.../hadoop/hbase/security/access/AccessController.java | 7 ---
.../security/visibility/VisibilityController.java | 7 ---
.../hbase/coprocessor/TestRegionCoprocessorHost.java | 1 +
5 files changed, 17 insertions(+), 32 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/constraint/ConstraintProcessor.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/constraint/ConstraintProcessor.java
index 6aa5d97..b0a04c5 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/constraint/ConstraintProcessor.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/constraint/ConstraintProcessor.java
@@ -22,20 +22,19 @@ import java.util.ArrayList;
import java.util.List;
import java.util.Optional;
-import org.apache.yetus.audience.InterfaceAudience;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import org.apache.hadoop.hbase.Cell;
import org.apache.hadoop.hbase.CoprocessorEnvironment;
-import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Durability;
+import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.TableDescriptor;
import org.apache.hadoop.hbase.coprocessor.ObserverContext;
import org.apache.hadoop.hbase.coprocessor.RegionCoprocessor;
import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
import org.apache.hadoop.hbase.coprocessor.RegionObserver;
-import org.apache.hadoop.hbase.regionserver.InternalScanner;
import org.apache.hadoop.hbase.wal.WALEdit;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/***
* Processes multiple {@link Constraint Constraints} on a given table.
@@ -98,11 +97,4 @@ public class ConstraintProcessor implements
RegionCoprocessor, RegionObserver {
}
// if we made it here, then the Put is valid
}
-
- @Override
- public boolean postScannerFilterRow(final
ObserverContext e,
- final InternalScanner s, final Cell curRowCell, final boolean hasMore)
throws IOException {
-// 'default' in RegionObserver might do unnecessary copy for Off heap
backed Cells.
-return hasMore;
- }
}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
index 16fd332..2feb270 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
@@ -105,6 +105,13 @@ public class RegionCoprocessorHost
// optimization: no need to call postScannerFilterRow, if no coprocessor
implements it
private final boolean hasCustomPostScannerFilterRow;
+ /*
+ * Whether any configured CPs override postScannerFilterRow hook
+ */
+ public boolean hasCustomPostScannerFilterRow() {
+return hasCustomPostScannerFilterRow;
+ }
+
/**
*
* Encapsulation of the environment of each coprocessor
@@ -278,11 +285,10 @@ public class RegionCoprocessorHost
out: for (RegionCoprocessorEnvironment env: coprocEnvironments) {
if (env.getInstance() instanceof RegionObserver) {
Class clazz = env.getInstance().getClass();
-for(;;) {
- if (clazz == null) {
-// we must have directly implemented RegionObserver
-hasCustomPostScannerFilterRow = true;
-break out;
+for (;;) {
+ if (clazz == Object.class) {
+// we dont need to look postScannerFilterRow into Object class
+break; // break the inner loop
}
try {
clazz.getDeclaredMethod("postScannerFilterRow",
ObserverContext.class,
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
index ab8a7e7..d747e7f 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/A
[hbase] branch branch-2.2 updated: HBASE-25427 [branch-2.2] Fix the ruby problem for pre-commit (#2798)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a commit to branch branch-2.2 in repository https://gitbox.apache.org/repos/asf/hbase.git The following commit(s) were added to refs/heads/branch-2.2 by this push: new 4e75ac5 HBASE-25427 [branch-2.2] Fix the ruby problem for pre-commit (#2798) 4e75ac5 is described below commit 4e75ac561a42b99a98d81dc86ab78b3ace6dc5af Author: Guanghao Zhang AuthorDate: Mon Dec 21 16:05:48 2020 +0800 HBASE-25427 [branch-2.2] Fix the ruby problem for pre-commit (#2798) Signed-off-by: Duo Zhang --- dev-support/docker/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev-support/docker/Dockerfile b/dev-support/docker/Dockerfile index 5c27ced..90bcc23 100644 --- a/dev-support/docker/Dockerfile +++ b/dev-support/docker/Dockerfile @@ -17,7 +17,7 @@ # Dockerfile for installing the necessary dependencies for building Hadoop. # See BUILDING.txt. -FROM maven:3.5-jdk-8 +FROM maven:3.6.3-jdk-8 RUN apt-get -q update && apt-get -q install --no-install-recommends -y \ git \
[hbase] branch branch-2 updated: HBASE-25324 Remove unnecessary array to list conversion in SplitLogManager (#2703)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-2 by this push:
new e81f49c HBASE-25324 Remove unnecessary array to list conversion in
SplitLogManager (#2703)
e81f49c is described below
commit e81f49c918ee9e45bc88dd714e5d8db55a013d91
Author: Qi Yu
AuthorDate: Fri Nov 27 08:20:24 2020 +0800
HBASE-25324 Remove unnecessary array to list conversion in SplitLogManager
(#2703)
Signed-off-by: Guanghao Zhang
Signed-off-by: Viraj Jasani
Signed-off-by: stack
---
.../org/apache/hadoop/hbase/master/SplitLogManager.java| 14 +++---
.../org/apache/hadoop/hbase/master/SplitWALManager.java| 9 -
.../main/java/org/apache/hadoop/hbase/wal/WALSplitter.java | 10 --
3 files changed, 15 insertions(+), 18 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
index 465a593..f628841 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
@@ -148,7 +148,7 @@ public class SplitLogManager {
return
server.getCoordinatedStateManager().getSplitLogManagerCoordination();
}
- private FileStatus[] getFileList(List logDirs, PathFilter filter)
throws IOException {
+ private List getFileList(List logDirs, PathFilter filter)
throws IOException {
return getFileList(conf, logDirs, filter);
}
@@ -163,7 +163,7 @@ public class SplitLogManager {
* {@link org.apache.hadoop.hbase.wal.WALSplitter#split(Path, Path, Path,
FileSystem,
* Configuration, org.apache.hadoop.hbase.wal.WALFactory)} for tests.
*/
- public static FileStatus[] getFileList(final Configuration conf, final
List logDirs,
+ public static List getFileList(final Configuration conf, final
List logDirs,
final PathFilter filter)
throws IOException {
List fileStatus = new ArrayList<>();
@@ -180,8 +180,8 @@ public class SplitLogManager {
Collections.addAll(fileStatus, logfiles);
}
}
-FileStatus[] a = new FileStatus[fileStatus.size()];
-return fileStatus.toArray(a);
+
+return fileStatus;
}
/**
@@ -239,11 +239,11 @@ public class SplitLogManager {
long totalSize = 0;
TaskBatch batch = null;
long startTime = 0;
-FileStatus[] logfiles = getFileList(logDirs, filter);
-if (logfiles.length != 0) {
+List logfiles = getFileList(logDirs, filter);
+if (!logfiles.isEmpty()) {
status.setStatus("Checking directory contents...");
SplitLogCounters.tot_mgr_log_split_batch_start.increment();
- LOG.info("Started splitting " + logfiles.length + " logs in " + logDirs +
+ LOG.info("Started splitting " + logfiles.size() + " logs in " + logDirs +
" for " + serverNames);
startTime = EnvironmentEdgeManager.currentTime();
batch = new TaskBatch();
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitWALManager.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitWALManager.java
index 48c19c2..aa91c84 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitWALManager.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitWALManager.java
@@ -43,7 +43,6 @@ import
org.apache.hadoop.hbase.procedure2.ProcedureSuspendedException;
import org.apache.hadoop.hbase.util.CommonFSUtils;
import org.apache.hadoop.hbase.wal.AbstractFSWALProvider;
import org.apache.hadoop.hbase.wal.WALSplitUtil;
-import org.apache.hbase.thirdparty.com.google.common.collect.Lists;
import org.apache.yetus.audience.InterfaceAudience;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -107,10 +106,10 @@ public class SplitWALManager {
public List getWALsToSplit(ServerName serverName, boolean
splitMeta)
throws IOException {
List logDirs =
master.getMasterWalManager().getLogDirs(Collections.singleton(serverName));
-FileStatus[] fileStatuses =
-SplitLogManager.getFileList(this.conf, logDirs, splitMeta ?
META_FILTER : NON_META_FILTER);
-LOG.info("{} WAL count={}, meta={}", serverName, fileStatuses.length,
splitMeta);
-return Lists.newArrayList(fileStatuses);
+List fileStatuses =
+ SplitLogManager.getFileList(this.conf, logDirs, splitMeta ? META_FILTER
: NON_META_FILTER);
+LOG.info("{} WAL count={}, meta={}", serverName, fileStatuses.size(),
splitMeta);
+return fileStatuses;
}
private Path getWALSplitDir(ServerName serverName) {
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
b/hbase-server/src/main
[hbase] branch master updated: HBASE-25324 Remove unnecessary array to list conversion in SplitLogManager (#2703)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/master by this push:
new dc48071 HBASE-25324 Remove unnecessary array to list conversion in
SplitLogManager (#2703)
dc48071 is described below
commit dc48071e01e19539647a4d6261cec0d25198d81e
Author: Qi Yu
AuthorDate: Fri Nov 27 08:20:24 2020 +0800
HBASE-25324 Remove unnecessary array to list conversion in SplitLogManager
(#2703)
Signed-off-by: Guanghao Zhang
Signed-off-by: Viraj Jasani
Signed-off-by: stack
---
.../org/apache/hadoop/hbase/master/SplitLogManager.java| 14 +++---
.../org/apache/hadoop/hbase/master/SplitWALManager.java| 9 -
.../main/java/org/apache/hadoop/hbase/wal/WALSplitter.java | 10 --
3 files changed, 15 insertions(+), 18 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
index 465a593..f628841 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
@@ -148,7 +148,7 @@ public class SplitLogManager {
return
server.getCoordinatedStateManager().getSplitLogManagerCoordination();
}
- private FileStatus[] getFileList(List logDirs, PathFilter filter)
throws IOException {
+ private List getFileList(List logDirs, PathFilter filter)
throws IOException {
return getFileList(conf, logDirs, filter);
}
@@ -163,7 +163,7 @@ public class SplitLogManager {
* {@link org.apache.hadoop.hbase.wal.WALSplitter#split(Path, Path, Path,
FileSystem,
* Configuration, org.apache.hadoop.hbase.wal.WALFactory)} for tests.
*/
- public static FileStatus[] getFileList(final Configuration conf, final
List logDirs,
+ public static List getFileList(final Configuration conf, final
List logDirs,
final PathFilter filter)
throws IOException {
List fileStatus = new ArrayList<>();
@@ -180,8 +180,8 @@ public class SplitLogManager {
Collections.addAll(fileStatus, logfiles);
}
}
-FileStatus[] a = new FileStatus[fileStatus.size()];
-return fileStatus.toArray(a);
+
+return fileStatus;
}
/**
@@ -239,11 +239,11 @@ public class SplitLogManager {
long totalSize = 0;
TaskBatch batch = null;
long startTime = 0;
-FileStatus[] logfiles = getFileList(logDirs, filter);
-if (logfiles.length != 0) {
+List logfiles = getFileList(logDirs, filter);
+if (!logfiles.isEmpty()) {
status.setStatus("Checking directory contents...");
SplitLogCounters.tot_mgr_log_split_batch_start.increment();
- LOG.info("Started splitting " + logfiles.length + " logs in " + logDirs +
+ LOG.info("Started splitting " + logfiles.size() + " logs in " + logDirs +
" for " + serverNames);
startTime = EnvironmentEdgeManager.currentTime();
batch = new TaskBatch();
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitWALManager.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitWALManager.java
index 48c19c2..aa91c84 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitWALManager.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitWALManager.java
@@ -43,7 +43,6 @@ import
org.apache.hadoop.hbase.procedure2.ProcedureSuspendedException;
import org.apache.hadoop.hbase.util.CommonFSUtils;
import org.apache.hadoop.hbase.wal.AbstractFSWALProvider;
import org.apache.hadoop.hbase.wal.WALSplitUtil;
-import org.apache.hbase.thirdparty.com.google.common.collect.Lists;
import org.apache.yetus.audience.InterfaceAudience;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -107,10 +106,10 @@ public class SplitWALManager {
public List getWALsToSplit(ServerName serverName, boolean
splitMeta)
throws IOException {
List logDirs =
master.getMasterWalManager().getLogDirs(Collections.singleton(serverName));
-FileStatus[] fileStatuses =
-SplitLogManager.getFileList(this.conf, logDirs, splitMeta ?
META_FILTER : NON_META_FILTER);
-LOG.info("{} WAL count={}, meta={}", serverName, fileStatuses.length,
splitMeta);
-return Lists.newArrayList(fileStatuses);
+List fileStatuses =
+ SplitLogManager.getFileList(this.conf, logDirs, splitMeta ? META_FILTER
: NON_META_FILTER);
+LOG.info("{} WAL count={}, meta={}", serverName, fileStatuses.size(),
splitMeta);
+return fileStatuses;
}
private Path getWALSplitDir(ServerName serverName) {
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
b/hbase-server/src/main
[hbase] branch branch-2.2 updated: HBASE-25323 Fix potential NPE when the zookeeper path of RegionServerTracker does not exist when start (#2702)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch branch-2.2
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-2.2 by this push:
new b710fbd HBASE-25323 Fix potential NPE when the zookeeper path of
RegionServerTracker does not exist when start (#2702)
b710fbd is described below
commit b710fbd243bf306f2f84467b89caafb3278e5106
Author: Qi Yu
AuthorDate: Wed Nov 25 15:13:09 2020 +0800
HBASE-25323 Fix potential NPE when the zookeeper path of
RegionServerTracker does not exist when start (#2702)
Signed-off-by: Duo Zhang
Signed-off-by: Guanghao Zhang
---
.../hadoop/hbase/master/RegionServerTracker.java | 28 +-
.../org/apache/hadoop/hbase/zookeeper/ZKUtil.java | 5 ++--
2 files changed, 19 insertions(+), 14 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionServerTracker.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionServerTracker.java
index 9d33a21..336f9dc 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionServerTracker.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionServerTracker.java
@@ -19,6 +19,7 @@ package org.apache.hadoop.hbase.master;
import java.io.IOException;
import java.io.InterruptedIOException;
+import java.util.Collections;
import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
@@ -35,6 +36,7 @@ import org.apache.hadoop.hbase.zookeeper.ZKListener;
import org.apache.hadoop.hbase.zookeeper.ZKUtil;
import org.apache.hadoop.hbase.zookeeper.ZKWatcher;
import org.apache.hadoop.hbase.zookeeper.ZNodePaths;
+import
org.apache.hbase.thirdparty.org.apache.commons.collections4.CollectionUtils;
import org.apache.yetus.audience.InterfaceAudience;
import org.apache.zookeeper.KeeperException;
import org.slf4j.Logger;
@@ -129,21 +131,24 @@ public class RegionServerTracker extends ZKListener {
splittingServersFromWALDir.stream().filter(s ->
!deadServersFromPE.contains(s)).
forEach(s -> LOG.error("{} has no matching ServerCrashProcedure", s));
//create ServerNode for all possible live servers from wal directory
-liveServersFromWALDir.stream()
+liveServersFromWALDir
.forEach(sn ->
server.getAssignmentManager().getRegionStates().getOrCreateServer(sn));
watcher.registerListener(this);
synchronized (this) {
List servers =
ZKUtil.listChildrenAndWatchForNewChildren(watcher,
watcher.getZNodePaths().rsZNode);
- for (String n : servers) {
-Pair pair = getServerInfo(n);
-ServerName serverName = pair.getFirst();
-RegionServerInfo info = pair.getSecond();
-regionServers.add(serverName);
-ServerMetrics serverMetrics = info != null ?
ServerMetricsBuilder.of(serverName,
- VersionInfoUtil.getVersionNumber(info.getVersionInfo()),
- info.getVersionInfo().getVersion()) :
ServerMetricsBuilder.of(serverName);
-serverManager.checkAndRecordNewServer(serverName, serverMetrics);
+ if (null != servers) {
+for (String n : servers) {
+ Pair pair = getServerInfo(n);
+ ServerName serverName = pair.getFirst();
+ RegionServerInfo info = pair.getSecond();
+ regionServers.add(serverName);
+ ServerMetrics serverMetrics = info != null ?
+ServerMetricsBuilder.of(serverName,
VersionInfoUtil.getVersionNumber(info.getVersionInfo()),
+ info.getVersionInfo().getVersion()) :
+ServerMetricsBuilder.of(serverName);
+ serverManager.checkAndRecordNewServer(serverName, serverMetrics);
+}
}
serverManager.findDeadServersAndProcess(deadServersFromPE,
liveServersFromWALDir);
}
@@ -163,8 +168,9 @@ public class RegionServerTracker extends ZKListener {
server.abort("Unexpected zk exception getting RS nodes", e);
return;
}
-Set servers =
+Set servers = CollectionUtils.isEmpty(names) ?
Collections.emptySet() :
names.stream().map(ServerName::parseServerName).collect(Collectors.toSet());
+
for (Iterator iter = regionServers.iterator();
iter.hasNext();) {
ServerName sn = iter.next();
if (!servers.contains(sn)) {
diff --git
a/hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
b/hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
index 584cb5f..1b8d929 100644
---
a/hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
+++
b/hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
@@ -443,16 +443,15 @@ public final class ZKUtil {
} catch(KeeperException.NoNodeException ke) {
LOG.debug(zkw.prefix("Unable to list children of znode " + znode + " " +
"because nod
[hbase] branch branch-2.3 updated: HBASE-25323 Fix potential NPE when the zookeeper path of RegionServerTracker does not exist when start (#2702)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch branch-2.3
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-2.3 by this push:
new 6bf770c HBASE-25323 Fix potential NPE when the zookeeper path of
RegionServerTracker does not exist when start (#2702)
6bf770c is described below
commit 6bf770c1f403aec453cf73bccaf587bf7065462b
Author: Qi Yu
AuthorDate: Wed Nov 25 15:13:09 2020 +0800
HBASE-25323 Fix potential NPE when the zookeeper path of
RegionServerTracker does not exist when start (#2702)
Signed-off-by: Duo Zhang
Signed-off-by: Guanghao Zhang
---
.../hadoop/hbase/master/RegionServerTracker.java | 28 +-
.../org/apache/hadoop/hbase/zookeeper/ZKUtil.java | 5 ++--
2 files changed, 19 insertions(+), 14 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionServerTracker.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionServerTracker.java
index 9d33a21..336f9dc 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionServerTracker.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionServerTracker.java
@@ -19,6 +19,7 @@ package org.apache.hadoop.hbase.master;
import java.io.IOException;
import java.io.InterruptedIOException;
+import java.util.Collections;
import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
@@ -35,6 +36,7 @@ import org.apache.hadoop.hbase.zookeeper.ZKListener;
import org.apache.hadoop.hbase.zookeeper.ZKUtil;
import org.apache.hadoop.hbase.zookeeper.ZKWatcher;
import org.apache.hadoop.hbase.zookeeper.ZNodePaths;
+import
org.apache.hbase.thirdparty.org.apache.commons.collections4.CollectionUtils;
import org.apache.yetus.audience.InterfaceAudience;
import org.apache.zookeeper.KeeperException;
import org.slf4j.Logger;
@@ -129,21 +131,24 @@ public class RegionServerTracker extends ZKListener {
splittingServersFromWALDir.stream().filter(s ->
!deadServersFromPE.contains(s)).
forEach(s -> LOG.error("{} has no matching ServerCrashProcedure", s));
//create ServerNode for all possible live servers from wal directory
-liveServersFromWALDir.stream()
+liveServersFromWALDir
.forEach(sn ->
server.getAssignmentManager().getRegionStates().getOrCreateServer(sn));
watcher.registerListener(this);
synchronized (this) {
List servers =
ZKUtil.listChildrenAndWatchForNewChildren(watcher,
watcher.getZNodePaths().rsZNode);
- for (String n : servers) {
-Pair pair = getServerInfo(n);
-ServerName serverName = pair.getFirst();
-RegionServerInfo info = pair.getSecond();
-regionServers.add(serverName);
-ServerMetrics serverMetrics = info != null ?
ServerMetricsBuilder.of(serverName,
- VersionInfoUtil.getVersionNumber(info.getVersionInfo()),
- info.getVersionInfo().getVersion()) :
ServerMetricsBuilder.of(serverName);
-serverManager.checkAndRecordNewServer(serverName, serverMetrics);
+ if (null != servers) {
+for (String n : servers) {
+ Pair pair = getServerInfo(n);
+ ServerName serverName = pair.getFirst();
+ RegionServerInfo info = pair.getSecond();
+ regionServers.add(serverName);
+ ServerMetrics serverMetrics = info != null ?
+ServerMetricsBuilder.of(serverName,
VersionInfoUtil.getVersionNumber(info.getVersionInfo()),
+ info.getVersionInfo().getVersion()) :
+ServerMetricsBuilder.of(serverName);
+ serverManager.checkAndRecordNewServer(serverName, serverMetrics);
+}
}
serverManager.findDeadServersAndProcess(deadServersFromPE,
liveServersFromWALDir);
}
@@ -163,8 +168,9 @@ public class RegionServerTracker extends ZKListener {
server.abort("Unexpected zk exception getting RS nodes", e);
return;
}
-Set servers =
+Set servers = CollectionUtils.isEmpty(names) ?
Collections.emptySet() :
names.stream().map(ServerName::parseServerName).collect(Collectors.toSet());
+
for (Iterator iter = regionServers.iterator();
iter.hasNext();) {
ServerName sn = iter.next();
if (!servers.contains(sn)) {
diff --git
a/hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
b/hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
index 19d11d0..981265e 100644
---
a/hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
+++
b/hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
@@ -447,16 +447,15 @@ public final class ZKUtil {
} catch(KeeperException.NoNodeException ke) {
LOG.debug(zkw.prefix("Unable to list children of znode " + znode + " " +
"because nod
[hbase] 01/02: HBASE-25325 Remove unused class ClusterSchemaException (#2704)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 6f07efb971d237f1c9f4d5209a878126104dd48e
Author: Qi Yu
AuthorDate: Wed Nov 25 15:09:40 2020 +0800
HBASE-25325 Remove unused class ClusterSchemaException (#2704)
Signed-off-by: Guanghao Zhang
Signed-off-by: Duo Zhang
Signed-off-by: Viraj Jasani
---
.../apache/hadoop/hbase/master/ClusterSchema.java | 14 +++-
.../hbase/master/ClusterSchemaException.java | 37 --
2 files changed, 5 insertions(+), 46 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ClusterSchema.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ClusterSchema.java
index 56a1f33..0f7153b 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ClusterSchema.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ClusterSchema.java
@@ -18,10 +18,10 @@
package org.apache.hadoop.hbase.master;
import java.io.IOException;
-import java.io.InterruptedIOException;
import java.util.List;
import org.apache.hadoop.hbase.NamespaceDescriptor;
+import org.apache.hadoop.hbase.ServiceNotRunningException;
import org.apache.hadoop.hbase.master.procedure.ProcedurePrepareLatch;
import org.apache.yetus.audience.InterfaceAudience;
import org.apache.hadoop.hbase.util.NonceKey;
@@ -82,8 +82,7 @@ public interface ClusterSchema {
* @param nonceKey A unique identifier for this operation from the client or
process.
* @param latch A latch to block on for precondition validation
* @return procedure id
- * @throws IOException Throws {@link ClusterSchemaException} and {@link
InterruptedIOException}
- *as well as {@link IOException}
+ * @throws IOException if service is not running see {@link
ServiceNotRunningException}
*/
long createNamespace(NamespaceDescriptor namespaceDescriptor, NonceKey
nonceKey, ProcedurePrepareLatch latch)
throws IOException;
@@ -93,8 +92,7 @@ public interface ClusterSchema {
* @param nonceKey A unique identifier for this operation from the client or
process.
* @param latch A latch to block on for precondition validation
* @return procedure id
- * @throws IOException Throws {@link ClusterSchemaException} and {@link
InterruptedIOException}
- *as well as {@link IOException}
+ * @throws IOException if service is not running see {@link
ServiceNotRunningException}
*/
long modifyNamespace(NamespaceDescriptor descriptor, NonceKey nonceKey,
ProcedurePrepareLatch latch)
throws IOException;
@@ -105,8 +103,7 @@ public interface ClusterSchema {
* @param nonceKey A unique identifier for this operation from the client or
process.
* @param latch A latch to block on for precondition validation
* @return procedure id
- * @throws IOException Throws {@link ClusterSchemaException} and {@link
InterruptedIOException}
- *as well as {@link IOException}
+ * @throws IOException if service is not running see {@link
ServiceNotRunningException}
*/
long deleteNamespace(String name, NonceKey nonceKey, ProcedurePrepareLatch
latch)
throws IOException;
@@ -115,8 +112,7 @@ public interface ClusterSchema {
* Get a Namespace
* @param name Name of the Namespace
* @return Namespace descriptor for name
- * @throws IOException Throws {@link ClusterSchemaException} and {@link
InterruptedIOException}
- *as well as {@link IOException}
+ * @throws IOException if namespace does not exist
*/
// No Future here because presumption is that the request will go against
cached metadata so
// return immediately -- no need of running a Procedure.
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ClusterSchemaException.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ClusterSchemaException.java
deleted file mode 100644
index e4d7736..000
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ClusterSchemaException.java
+++ /dev/null
@@ -1,37 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the Li
[hbase] 02/02: HBASE-25323 Fix potential NPE when the zookeeper path of RegionServerTracker does not exist when start (#2702)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 3dd425abfa7a2c458a54b6de1f502e33bcd44e20
Author: Qi Yu
AuthorDate: Wed Nov 25 15:13:09 2020 +0800
HBASE-25323 Fix potential NPE when the zookeeper path of
RegionServerTracker does not exist when start (#2702)
Signed-off-by: Duo Zhang
Signed-off-by: Guanghao Zhang
---
.../hadoop/hbase/master/RegionServerTracker.java | 28 +-
.../hbase/zookeeper/MasterAddressTracker.java | 2 +-
.../org/apache/hadoop/hbase/zookeeper/ZKUtil.java | 5 ++--
3 files changed, 20 insertions(+), 15 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionServerTracker.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionServerTracker.java
index 9d33a21..336f9dc 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionServerTracker.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionServerTracker.java
@@ -19,6 +19,7 @@ package org.apache.hadoop.hbase.master;
import java.io.IOException;
import java.io.InterruptedIOException;
+import java.util.Collections;
import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
@@ -35,6 +36,7 @@ import org.apache.hadoop.hbase.zookeeper.ZKListener;
import org.apache.hadoop.hbase.zookeeper.ZKUtil;
import org.apache.hadoop.hbase.zookeeper.ZKWatcher;
import org.apache.hadoop.hbase.zookeeper.ZNodePaths;
+import
org.apache.hbase.thirdparty.org.apache.commons.collections4.CollectionUtils;
import org.apache.yetus.audience.InterfaceAudience;
import org.apache.zookeeper.KeeperException;
import org.slf4j.Logger;
@@ -129,21 +131,24 @@ public class RegionServerTracker extends ZKListener {
splittingServersFromWALDir.stream().filter(s ->
!deadServersFromPE.contains(s)).
forEach(s -> LOG.error("{} has no matching ServerCrashProcedure", s));
//create ServerNode for all possible live servers from wal directory
-liveServersFromWALDir.stream()
+liveServersFromWALDir
.forEach(sn ->
server.getAssignmentManager().getRegionStates().getOrCreateServer(sn));
watcher.registerListener(this);
synchronized (this) {
List servers =
ZKUtil.listChildrenAndWatchForNewChildren(watcher,
watcher.getZNodePaths().rsZNode);
- for (String n : servers) {
-Pair pair = getServerInfo(n);
-ServerName serverName = pair.getFirst();
-RegionServerInfo info = pair.getSecond();
-regionServers.add(serverName);
-ServerMetrics serverMetrics = info != null ?
ServerMetricsBuilder.of(serverName,
- VersionInfoUtil.getVersionNumber(info.getVersionInfo()),
- info.getVersionInfo().getVersion()) :
ServerMetricsBuilder.of(serverName);
-serverManager.checkAndRecordNewServer(serverName, serverMetrics);
+ if (null != servers) {
+for (String n : servers) {
+ Pair pair = getServerInfo(n);
+ ServerName serverName = pair.getFirst();
+ RegionServerInfo info = pair.getSecond();
+ regionServers.add(serverName);
+ ServerMetrics serverMetrics = info != null ?
+ServerMetricsBuilder.of(serverName,
VersionInfoUtil.getVersionNumber(info.getVersionInfo()),
+ info.getVersionInfo().getVersion()) :
+ServerMetricsBuilder.of(serverName);
+ serverManager.checkAndRecordNewServer(serverName, serverMetrics);
+}
}
serverManager.findDeadServersAndProcess(deadServersFromPE,
liveServersFromWALDir);
}
@@ -163,8 +168,9 @@ public class RegionServerTracker extends ZKListener {
server.abort("Unexpected zk exception getting RS nodes", e);
return;
}
-Set servers =
+Set servers = CollectionUtils.isEmpty(names) ?
Collections.emptySet() :
names.stream().map(ServerName::parseServerName).collect(Collectors.toSet());
+
for (Iterator iter = regionServers.iterator();
iter.hasNext();) {
ServerName sn = iter.next();
if (!servers.contains(sn)) {
diff --git
a/hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MasterAddressTracker.java
b/hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MasterAddressTracker.java
index 3206238..133d966 100644
---
a/hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MasterAddressTracker.java
+++
b/hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MasterAddressTracker.java
@@ -297,7 +297,7 @@ public class MasterAddressTracker extends ZKNodeTracker {
public static List getBackupMastersAndRenewWatch(
ZKWatcher zkw) throws InterruptedIOException {
// Build Set of backup masters from ZK nodes
-List backupMasterStrings = Collections.emptyList();
+List backupMasterStrings = null;
try
[hbase] branch branch-2 updated (8e3727e -> 3dd425a)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch branch-2 in repository https://gitbox.apache.org/repos/asf/hbase.git. from 8e3727e HBASE-25281 Bulkload split hfile too many times due to unreasonable split point (#2692) new 6f07efb HBASE-25325 Remove unused class ClusterSchemaException (#2704) new 3dd425a HBASE-25323 Fix potential NPE when the zookeeper path of RegionServerTracker does not exist when start (#2702) The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/hadoop/hbase/master/ClusterSchema.java | 14 +++- .../hbase/master/ClusterSchemaException.java | 37 -- .../hadoop/hbase/master/RegionServerTracker.java | 28 +--- .../hbase/zookeeper/MasterAddressTracker.java | 2 +- .../org/apache/hadoop/hbase/zookeeper/ZKUtil.java | 5 ++- 5 files changed, 25 insertions(+), 61 deletions(-) delete mode 100644 hbase-server/src/main/java/org/apache/hadoop/hbase/master/ClusterSchemaException.java
[hbase] branch branch-2 updated: HBASE-25281 Bulkload split hfile too many times due to unreasonable split point (#2692)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-2 by this push:
new 8e3727e HBASE-25281 Bulkload split hfile too many times due to
unreasonable split point (#2692)
8e3727e is described below
commit 8e3727ea06e85f37247f060598a54d37b303f21a
Author: niuyulin
AuthorDate: Wed Nov 25 15:27:09 2020 +0800
HBASE-25281 Bulkload split hfile too many times due to unreasonable split
point (#2692)
Signed-off-by: Guanghao Zhang
---
.../hadoop/hbase/tool/LoadIncrementalHFiles.java | 92 ++
.../TestLoadIncrementalHFilesSplitRecovery.java| 28 +++
2 files changed, 85 insertions(+), 35 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/LoadIncrementalHFiles.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/LoadIncrementalHFiles.java
index ddc857c..866faf2 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/LoadIncrementalHFiles.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/LoadIncrementalHFiles.java
@@ -723,6 +723,45 @@ public class LoadIncrementalHFiles extends Configured
implements Tool {
}
/**
+ * @param startEndKeys the start/end keys of regions belong to this table,
the list in ascending
+ * order by start key
+ * @param key the key need to find which region belong to
+ * @return region index
+ */
+ private int getRegionIndex(final Pair startEndKeys,
byte[] key) {
+int idx = Arrays.binarySearch(startEndKeys.getFirst(), key,
Bytes.BYTES_COMPARATOR);
+if (idx < 0) {
+ // not on boundary, returns -(insertion index). Calculate region it
+ // would be in.
+ idx = -(idx + 1) - 1;
+}
+return idx;
+ }
+
+ /**
+ * we can consider there is a region hole in following conditions. 1) if idx
< 0,then first
+ * region info is lost. 2) if the endkey of a region is not equal to the
startkey of the next
+ * region. 3) if the endkey of the last region is not empty.
+ */
+ private void checkRegionIndexValid(int idx, final Pair
startEndKeys,
+TableName tableName) throws IOException {
+if (idx < 0) {
+ throw new IOException("The first region info for table " + tableName +
+" can't be found in hbase:meta.Please use hbck tool to fix it first.");
+} else if ((idx == startEndKeys.getFirst().length - 1) &&
+ !Bytes.equals(startEndKeys.getSecond()[idx],
HConstants.EMPTY_BYTE_ARRAY)) {
+ throw new IOException("The last region info for table " + tableName +
+" can't be found in hbase:meta.Please use hbck tool to fix it first.");
+} else if (idx + 1 < startEndKeys.getFirst().length &&
+ !(Bytes.compareTo(startEndKeys.getSecond()[idx],
+startEndKeys.getFirst()[idx + 1]) == 0)) {
+ throw new IOException("The endkey of one region for table " + tableName +
+" is not equal to the startkey of the next region in hbase:meta." +
+"Please use hbck tool to fix it first.");
+}
+ }
+
+ /**
* Attempt to assign the given load queue item into its target region group.
If the hfile boundary
* no longer fits into a region, physically splits the hfile such that the
new bottom half will
* fit and returns the list of LQI's corresponding to the resultant hfiles.
@@ -745,8 +784,8 @@ public class LoadIncrementalHFiles extends Configured
implements Tool {
return new Pair<>(null, hfilePath.getName());
}
-LOG.info("Trying to load hfile=" + hfilePath + " first=" +
first.map(Bytes::toStringBinary) +
-" last=" + last.map(Bytes::toStringBinary));
+LOG.info("Trying to load hfile=" + hfilePath + " first=" +
first.map(Bytes::toStringBinary)
++ " last=" + last.map(Bytes::toStringBinary));
if (!first.isPresent() || !last.isPresent()) {
assert !first.isPresent() && !last.isPresent();
// TODO what if this is due to a bad HFile?
@@ -754,47 +793,30 @@ public class LoadIncrementalHFiles extends Configured
implements Tool {
return null;
}
if (Bytes.compareTo(first.get(), last.get()) > 0) {
- throw new IllegalArgumentException("Invalid range: " +
Bytes.toStringBinary(first.get()) +
- " > " + Bytes.toStringBinary(last.get()));
-}
-int idx = Arrays.binarySearch(startEndKeys.getFirst(), first.get(),
Bytes.BYTES_COMPARATOR);
-if (idx < 0) {
- // not on boundary, returns -(insertion index). Calculate region it
- // would be in.
- idx = -(idx + 1) - 1;
-}
-int indexForCallable = idx;
-
-/**
- * we can consider there is a region hole in following cond
[hbase] branch master updated (984d578 -> 403756a)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hbase.git. from 984d578 HBASE-25323 Fix potential NPE when the zookeeper path of RegionServerTracker does not exist when start (#2702) add 403756a HBASE-25282 Remove processingServers in DeadServer as we can get this… (#2657) No new revisions were added by this update. Summary of changes: .../org/apache/hadoop/hbase/master/DeadServer.java | 52 -- .../hadoop/hbase/master/MasterRpcServices.java | 9 .../apache/hadoop/hbase/master/ServerManager.java | 5 ++- .../master/procedure/ServerCrashProcedure.java | 2 - .../apache/hadoop/hbase/TestRegionRebalancing.java | 2 +- .../apache/hadoop/hbase/master/TestDeadServer.java | 20 ++--- .../hadoop/hbase/master/TestRollingRestart.java| 2 +- .../hadoop/hbase/master/procedure/TestHBCKSCP.java | 1 - .../TestSyncReplicationStandbyKillRS.java | 3 +- 9 files changed, 19 insertions(+), 77 deletions(-)
[hbase] branch master updated (31d880b -> 984d578)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hbase.git. from 31d880b HBASE-25325 Remove unused class ClusterSchemaException (#2704) add 984d578 HBASE-25323 Fix potential NPE when the zookeeper path of RegionServerTracker does not exist when start (#2702) No new revisions were added by this update. Summary of changes: .../hadoop/hbase/master/RegionServerTracker.java | 28 +- .../hbase/zookeeper/MasterAddressTracker.java | 2 +- .../org/apache/hadoop/hbase/zookeeper/ZKUtil.java | 5 ++-- 3 files changed, 20 insertions(+), 15 deletions(-)
[hbase] branch master updated (4affae5 -> 31d880b)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hbase.git. from 4affae5 HBASE-25213 Should request Compaction when bulkLoadHFiles is done (addendum) (#2701) add 31d880b HBASE-25325 Remove unused class ClusterSchemaException (#2704) No new revisions were added by this update. Summary of changes: .../apache/hadoop/hbase/master/ClusterSchema.java | 14 +++- .../hbase/master/ClusterSchemaException.java | 37 -- 2 files changed, 5 insertions(+), 46 deletions(-) delete mode 100644 hbase-server/src/main/java/org/apache/hadoop/hbase/master/ClusterSchemaException.java
[hbase] branch branch-2 updated: HBASE-25213 Should request Compaction when bulkLoadHFiles is done (addendum) (#2701)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-2 by this push:
new d523d75 HBASE-25213 Should request Compaction when bulkLoadHFiles is
done (addendum) (#2701)
d523d75 is described below
commit d523d758af94d546e4af0bd0f2631e285c67820e
Author: niuyulin
AuthorDate: Wed Nov 25 07:57:18 2020 +0800
HBASE-25213 Should request Compaction when bulkLoadHFiles is done
(addendum) (#2701)
Signed-off-by: Guanghao Zhang
---
.../hbase/quotas/TestLowLatencySpaceQuotas.java | 20 +++-
1 file changed, 15 insertions(+), 5 deletions(-)
diff --git
a/hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestLowLatencySpaceQuotas.java
b/hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestLowLatencySpaceQuotas.java
index 7fad94a..994e182 100644
---
a/hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestLowLatencySpaceQuotas.java
+++
b/hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestLowLatencySpaceQuotas.java
@@ -22,6 +22,7 @@ import static org.junit.Assert.assertTrue;
import java.util.Collections;
import java.util.List;
import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
@@ -29,6 +30,7 @@ import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.HBaseClassTestRule;
import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.ServerName;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Admin;
import org.apache.hadoop.hbase.client.ClientServiceCallable;
@@ -213,6 +215,8 @@ public class TestLowLatencySpaceQuotas {
QuotaSettings settings = QuotaSettingsFactory.limitTableSpace(
tn, SpaceQuotaHelperForTests.ONE_GIGABYTE,
SpaceViolationPolicy.NO_INSERTS);
admin.setQuota(settings);
+admin.compactionSwitch(false,
+
admin.getRegionServers().stream().map(ServerName::toString).collect(Collectors.toList()));
ClientServiceCallable callable = helper.generateFileToLoad(tn, 3,
550);
// Make sure the files are about as long as we expect
@@ -233,11 +237,17 @@ public class TestLowLatencySpaceQuotas {
caller.callWithRetries(callable, Integer.MAX_VALUE);
final long finalTotalSize = totalSize;
-TEST_UTIL.waitFor(30 * 1000, 500, new SpaceQuotaSnapshotPredicate(conn,
tn) {
- @Override boolean evaluate(SpaceQuotaSnapshot snapshot) throws Exception
{
-return snapshot.getUsage() >= finalTotalSize;
- }
-});
+try {
+ TEST_UTIL.waitFor(30 * 1000, 500, new SpaceQuotaSnapshotPredicate(conn,
tn) {
+@Override
+boolean evaluate(SpaceQuotaSnapshot snapshot) throws Exception {
+ return snapshot.getUsage() >= finalTotalSize;
+}
+ });
+} finally {
+ admin.compactionSwitch(true,
+
admin.getRegionServers().stream().map(ServerName::toString).collect(Collectors.toList()));
+}
}
@Test
[hbase] branch master updated (fc33137 -> 4affae5)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hbase.git. from fc33137 HBASE-24268 REST and Thrift server do not handle the "doAs" parameter case insensitively add 4affae5 HBASE-25213 Should request Compaction when bulkLoadHFiles is done (addendum) (#2701) No new revisions were added by this update. Summary of changes: .../hbase/quotas/TestLowLatencySpaceQuotas.java| 22 +++--- 1 file changed, 15 insertions(+), 7 deletions(-)
[hbase] branch master updated: HBASE-24654 Allow unset table's rsgroup (#1994)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/master by this push:
new 1cd8f3c HBASE-24654 Allow unset table's rsgroup (#1994)
1cd8f3c is described below
commit 1cd8f3cf94c0b2dc6999fa17b4f3d32e16d37ac7
Author: XinSun
AuthorDate: Mon Nov 23 17:22:37 2020 +0800
HBASE-24654 Allow unset table's rsgroup (#1994)
Signed-off-by: Guanghao Zhang
Signed-off-by: Duo Zhang
---
.../hbase/client/TableDescriptorBuilder.java | 2 +-
.../hbase/client/TestTableDescriptorBuilder.java | 12
.../hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java | 35 ++
.../src/test/ruby/shell/rsgroup_shell_test.rb | 18 +++
4 files changed, 66 insertions(+), 1 deletion(-)
diff --git
a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java
b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java
index c611a21..d2cfff5 100644
---
a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java
+++
b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java
@@ -541,7 +541,7 @@ public class TableDescriptorBuilder {
}
public TableDescriptorBuilder setRegionServerGroup(String group) {
-desc.setValue(RSGROUP_KEY, new Bytes(Bytes.toBytes(group)));
+desc.setValue(RSGROUP_KEY, group);
return this;
}
diff --git
a/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestTableDescriptorBuilder.java
b/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestTableDescriptorBuilder.java
index 89d740c..c29c835 100644
---
a/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestTableDescriptorBuilder.java
+++
b/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestTableDescriptorBuilder.java
@@ -19,6 +19,7 @@ package org.apache.hadoop.hbase.client;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNull;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
@@ -27,6 +28,7 @@ import java.util.regex.Pattern;
import org.apache.hadoop.hbase.HBaseClassTestRule;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.exceptions.DeserializationException;
+import org.apache.hadoop.hbase.rsgroup.RSGroupInfo;
import org.apache.hadoop.hbase.testclassification.MiscTests;
import org.apache.hadoop.hbase.testclassification.SmallTests;
import org.apache.hadoop.hbase.util.BuilderStyleTest;
@@ -291,4 +293,14 @@ public class TestTableDescriptorBuilder {
"{TABLE_ATTRIBUTES => {DURABILITY => 'ASYNC_WAL'}}, {NAME => 'cf',
BLOCKSIZE => '1000'}",
htd.toStringCustomizedValues());
}
+
+ @Test
+ public void testGetSetRegionServerGroup() {
+String groupName = name.getMethodName();
+TableDescriptor htd =
TableDescriptorBuilder.newBuilder(TableName.valueOf(name.getMethodName()))
+.setRegionServerGroup(groupName).build();
+assertEquals(htd.getValue(RSGroupInfo.TABLE_DESC_PROP_GROUP), groupName);
+htd =
TableDescriptorBuilder.newBuilder(htd).setRegionServerGroup(null).build();
+assertNull(htd.getValue(RSGroupInfo.TABLE_DESC_PROP_GROUP));
+ }
}
diff --git
a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java
b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java
index bf7a2fd..87cb76e 100644
---
a/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java
+++
b/hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java
@@ -586,4 +586,39 @@ public class TestRSGroupsAdmin1 extends TestRSGroupsBase {
}
}
+
+ @Test
+ public void testTableConstraint() throws Exception {
+String prefix = name.getMethodName();
+String ns = prefix + "_ns";
+TableName tableName = TableName.valueOf(ns + ":" + "t");
+String nsGroup = prefix + "_nsg";
+String tableGroup = prefix + "_tg";
+addGroup(nsGroup, 1);
+addGroup(tableGroup, 1);
+ADMIN.createNamespace(NamespaceDescriptor.create(ns).build());
+TEST_UTIL.createTable(tableName, "C");
+TEST_UTIL.waitTableAvailable(tableName);
+assertEquals(ADMIN.getRSGroup(tableName).getName(),
RSGroupInfo.DEFAULT_GROUP);
+// set table's rsgroup
+TableDescriptor td =
TableDescriptorBuilder.newBuilder(ADMIN.getDescriptor(tableName))
+.setRegionServerGroup(tableGroup).build();
+ADMIN.modifyTable(td);
+TEST_UTIL.waitUntilNoRegionsInTransition();
+assertEquals(ADMIN.getRSGroup(tableName).getName(), tableGroup);
+// set namespace&#
[hbase] branch branch-2.2 updated: HBASE-25311 UI throws NPE (#2688)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch branch-2.2
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-2.2 by this push:
new 59c602b HBASE-25311 UI throws NPE (#2688)
59c602b is described below
commit 59c602b08015e3983d5508679ae4d81236ed6941
Author: Bo Cui
AuthorDate: Mon Nov 23 12:07:42 2020 +0800
HBASE-25311 UI throws NPE (#2688)
Signed-off-by: Guanghao Zhang
Signed-off-by: Wellington Chevreuil
Signed-off-by: Viraj Jasani
---
.../src/main/java/org/apache/hadoop/hbase/master/HMaster.java | 11 +--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
index d463bc0..95a2e5c 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
@@ -3660,8 +3660,15 @@ public class HMaster extends HRegionServer implements
MasterServices {
List replicationLoadSources =
getServerManager().getLoad(serverName).getReplicationLoadSourceList();
for (ReplicationLoadSource replicationLoadSource :
replicationLoadSources) {
-replicationLoadSourceMap.get(replicationLoadSource.getPeerID())
-.add(new Pair<>(serverName, replicationLoadSource));
+List>
replicationLoadSourceList =
+ replicationLoadSourceMap.get(replicationLoadSource.getPeerID());
+if (replicationLoadSourceList == null) {
+ LOG.debug("{} does not exist, but it exists "
++ "in znode(/hbase/replication/rs). when the rs restarts, peerId
is deleted, so "
++ "we just need to ignore it", replicationLoadSource.getPeerID());
+ continue;
+}
+replicationLoadSourceList.add(new Pair<>(serverName,
replicationLoadSource));
}
}
for (List> loads :
replicationLoadSourceMap.values()) {
[hbase] branch branch-2.3 updated: HBASE-25311 UI throws NPE (#2688)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch branch-2.3
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-2.3 by this push:
new 9d6261c HBASE-25311 UI throws NPE (#2688)
9d6261c is described below
commit 9d6261cc270e514149739c8047dcb238d519376d
Author: Bo Cui
AuthorDate: Mon Nov 23 12:07:42 2020 +0800
HBASE-25311 UI throws NPE (#2688)
Signed-off-by: Guanghao Zhang
Signed-off-by: Wellington Chevreuil
Signed-off-by: Viraj Jasani
---
.../src/main/java/org/apache/hadoop/hbase/master/HMaster.java | 11 +--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
index f758b8b..ac731a0 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
@@ -3822,8 +3822,15 @@ public class HMaster extends HRegionServer implements
MasterServices {
List replicationLoadSources =
getServerManager().getLoad(serverName).getReplicationLoadSourceList();
for (ReplicationLoadSource replicationLoadSource :
replicationLoadSources) {
-replicationLoadSourceMap.get(replicationLoadSource.getPeerID())
-.add(new Pair<>(serverName, replicationLoadSource));
+List>
replicationLoadSourceList =
+ replicationLoadSourceMap.get(replicationLoadSource.getPeerID());
+if (replicationLoadSourceList == null) {
+ LOG.debug("{} does not exist, but it exists "
++ "in znode(/hbase/replication/rs). when the rs restarts, peerId
is deleted, so "
++ "we just need to ignore it", replicationLoadSource.getPeerID());
+ continue;
+}
+replicationLoadSourceList.add(new Pair<>(serverName,
replicationLoadSource));
}
}
for (List> loads :
replicationLoadSourceMap.values()) {
[hbase] branch branch-2 updated: HBASE-25311 UI throws NPE (#2688)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-2 by this push:
new 0a1decd HBASE-25311 UI throws NPE (#2688)
0a1decd is described below
commit 0a1decdf1805654d864669d2ccaf8f9f93fe0714
Author: Bo Cui
AuthorDate: Mon Nov 23 12:07:42 2020 +0800
HBASE-25311 UI throws NPE (#2688)
Signed-off-by: Guanghao Zhang
Signed-off-by: Wellington Chevreuil
Signed-off-by: Viraj Jasani
---
.../src/main/java/org/apache/hadoop/hbase/master/HMaster.java | 11 +--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
index c60d138..9ab4355 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
@@ -3595,8 +3595,15 @@ public class HMaster extends HRegionServer implements
MasterServices {
List replicationLoadSources =
getServerManager().getLoad(serverName).getReplicationLoadSourceList();
for (ReplicationLoadSource replicationLoadSource :
replicationLoadSources) {
-replicationLoadSourceMap.get(replicationLoadSource.getPeerID())
-.add(new Pair<>(serverName, replicationLoadSource));
+List>
replicationLoadSourceList =
+ replicationLoadSourceMap.get(replicationLoadSource.getPeerID());
+if (replicationLoadSourceList == null) {
+ LOG.debug("{} does not exist, but it exists "
++ "in znode(/hbase/replication/rs). when the rs restarts, peerId
is deleted, so "
++ "we just need to ignore it", replicationLoadSource.getPeerID());
+ continue;
+}
+replicationLoadSourceList.add(new Pair<>(serverName,
replicationLoadSource));
}
}
for (List> loads :
replicationLoadSourceMap.values()) {
[hbase] branch master updated: HBASE-25311 UI throws NPE (#2688)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/master by this push:
new b0f20399e HBASE-25311 UI throws NPE (#2688)
b0f20399e is described below
commit b0f20399e7c2aab624e6e3eda6b1838b40c41faf
Author: Bo Cui
AuthorDate: Mon Nov 23 12:07:42 2020 +0800
HBASE-25311 UI throws NPE (#2688)
Signed-off-by: Guanghao Zhang
Signed-off-by: Wellington Chevreuil
Signed-off-by: Viraj Jasani
---
.../src/main/java/org/apache/hadoop/hbase/master/HMaster.java | 11 +--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
index bbd8c0e..7adbeb0 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
@@ -3621,8 +3621,15 @@ public class HMaster extends HRegionServer implements
MasterServices {
List replicationLoadSources =
getServerManager().getLoad(serverName).getReplicationLoadSourceList();
for (ReplicationLoadSource replicationLoadSource :
replicationLoadSources) {
-replicationLoadSourceMap.get(replicationLoadSource.getPeerID())
-.add(new Pair<>(serverName, replicationLoadSource));
+List>
replicationLoadSourceList =
+ replicationLoadSourceMap.get(replicationLoadSource.getPeerID());
+if (replicationLoadSourceList == null) {
+ LOG.debug("{} does not exist, but it exists "
++ "in znode(/hbase/replication/rs). when the rs restarts, peerId
is deleted, so "
++ "we just need to ignore it", replicationLoadSource.getPeerID());
+ continue;
+}
+replicationLoadSourceList.add(new Pair<>(serverName,
replicationLoadSource));
}
}
for (List> loads :
replicationLoadSourceMap.values()) {
[hbase] branch HBASE-24666 updated: HBASE-25113 [testing] HBaseCluster support ReplicationServer for UTs (#2662)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch HBASE-24666
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/HBASE-24666 by this push:
new 6cdd4f3 HBASE-25113 [testing] HBaseCluster support ReplicationServer
for UTs (#2662)
6cdd4f3 is described below
commit 6cdd4f3e56568191d4ab5e5c348c24f2ee32edf7
Author: XinSun
AuthorDate: Mon Nov 23 11:01:55 2020 +0800
HBASE-25113 [testing] HBaseCluster support ReplicationServer for UTs (#2662)
Signed-off-by: Guanghao Zhang
---
.../org/apache/hadoop/hbase/LocalHBaseCluster.java | 63 ++-
.../hbase/replication/HReplicationServer.java | 13
.../apache/hadoop/hbase/util/JVMClusterUtil.java | 57 +-
.../apache/hadoop/hbase/HBaseTestingUtility.java | 8 +--
.../org/apache/hadoop/hbase/MiniHBaseCluster.java | 70 ++
.../hadoop/hbase/StartMiniClusterOption.java | 24 ++--
.../replication/TestReplicationServerSink.java | 44 +++---
hbase-server/src/test/resources/hbase-site.xml | 7 +++
8 files changed, 242 insertions(+), 44 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java
index f4847b9..24b658f 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java
@@ -32,9 +32,11 @@ import org.apache.hadoop.hbase.client.TableDescriptor;
import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
import org.apache.hadoop.hbase.master.HMaster;
import org.apache.hadoop.hbase.regionserver.HRegionServer;
+import org.apache.hadoop.hbase.replication.HReplicationServer;
import org.apache.hadoop.hbase.security.User;
import org.apache.hadoop.hbase.util.JVMClusterUtil;
import org.apache.hadoop.hbase.util.JVMClusterUtil.RegionServerThread;
+import org.apache.hadoop.hbase.util.JVMClusterUtil.ReplicationServerThread;
import org.apache.hadoop.hbase.util.Threads;
import org.apache.yetus.audience.InterfaceAudience;
import org.slf4j.Logger;
@@ -60,7 +62,10 @@ import org.slf4j.LoggerFactory;
public class LocalHBaseCluster {
private static final Logger LOG =
LoggerFactory.getLogger(LocalHBaseCluster.class);
private final List masterThreads = new
CopyOnWriteArrayList<>();
- private final List regionThreads = new
CopyOnWriteArrayList<>();
+ private final List regionThreads =
+ new CopyOnWriteArrayList<>();
+ private final List
replicationThreads =
+ new CopyOnWriteArrayList<>();
private final static int DEFAULT_NO = 1;
/** local mode */
public static final String LOCAL = "local";
@@ -259,6 +264,26 @@ public class LocalHBaseCluster {
});
}
+ @SuppressWarnings("unchecked")
+ public JVMClusterUtil.ReplicationServerThread addReplicationServer(
+ Configuration config, final int index) throws IOException {
+// Create each replication server with its own Configuration instance so
each has
+// its Connection instance rather than share (see HBASE_INSTANCES down in
+// the guts of ConnectionManager).
+JVMClusterUtil.ReplicationServerThread rst =
+JVMClusterUtil.createReplicationServerThread(config, index);
+this.replicationThreads.add(rst);
+return rst;
+ }
+
+ public JVMClusterUtil.ReplicationServerThread addReplicationServer(
+ final Configuration config, final int index, User user)
+ throws IOException, InterruptedException {
+return user.runAs(
+(PrivilegedExceptionAction) () ->
addReplicationServer(config,
+index));
+ }
+
/**
* @param serverNumber
* @return region server
@@ -290,6 +315,40 @@ public class LocalHBaseCluster {
}
/**
+ * @param serverNumber replication server number
+ * @return replication server
+ */
+ public HReplicationServer getReplicationServer(int serverNumber) {
+return replicationThreads.get(serverNumber).getReplicationServer();
+ }
+
+ /**
+ * @return Read-only list of replication server threads.
+ */
+ public List getReplicationServers() {
+return Collections.unmodifiableList(this.replicationThreads);
+ }
+
+ /**
+ * @return List of running servers (Some servers may have been killed or
+ * aborted during lifetime of cluster; these servers are not included in
this
+ * list).
+ */
+ public List
getLiveReplicationServers() {
+List liveServers = new
ArrayList<>();
+List list = getReplicationServers();
+for (JVMClusterUtil.ReplicationServerThread rst: list) {
+ if (rst.isAlive()) {
+liveServers.add(rst);
+ }
+ else {
+LOG.info("Not alive {}", rst.getName());
+ }
+}
+return liveServers;
+ }
+
+ /**
* @return th
[hbase] branch master updated: HBASE-25281 Bulkload split hfile too many times due to unreasonable split point (#2667)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/master by this push:
new a307d70 HBASE-25281 Bulkload split hfile too many times due to
unreasonable split point (#2667)
a307d70 is described below
commit a307d706c8486c690644fdb0e4f3f0a4858f4c85
Author: niuyulin
AuthorDate: Mon Nov 23 10:07:19 2020 +0800
HBASE-25281 Bulkload split hfile too many times due to unreasonable split
point (#2667)
Signed-off-by: Guanghao Zhang
Signed-off-by: Wellington Chevreuil
---
.../hadoop/hbase/tool/BulkLoadHFilesTool.java | 96 +-
.../tool/TestBulkLoadHFilesSplitRecovery.java | 27 ++
2 files changed, 84 insertions(+), 39 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/BulkLoadHFilesTool.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/BulkLoadHFilesTool.java
index 6c62a3d..a39ecaf 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/BulkLoadHFilesTool.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/BulkLoadHFilesTool.java
@@ -616,6 +616,45 @@ public class BulkLoadHFilesTool extends Configured
implements BulkLoadHFiles, To
}
/**
+ * @param startEndKeys the start/end keys of regions belong to this table,
the list in ascending
+ * order by start key
+ * @param key the key need to find which region belong to
+ * @return region index
+ */
+ private int getRegionIndex(List> startEndKeys, byte[]
key) {
+int idx = Collections.binarySearch(startEndKeys, Pair.newPair(key,
HConstants.EMPTY_END_ROW),
+ (p1, p2) -> Bytes.compareTo(p1.getFirst(), p2.getFirst()));
+if (idx < 0) {
+ // not on boundary, returns -(insertion index). Calculate region it
+ // would be in.
+ idx = -(idx + 1) - 1;
+}
+return idx;
+ }
+
+ /**
+ * we can consider there is a region hole or overlap in following
conditions. 1) if idx < 0,then
+ * first region info is lost. 2) if the endkey of a region is not equal to
the startkey of the
+ * next region. 3) if the endkey of the last region is not empty.
+ */
+ private void checkRegionIndexValid(int idx, List>
startEndKeys,
+ TableName tableName) throws IOException {
+if (idx < 0) {
+ throw new IOException("The first region info for table " + tableName
+ + " can't be found in hbase:meta.Please use hbck tool to fix it
first.");
+} else if ((idx == startEndKeys.size() - 1)
+&& !Bytes.equals(startEndKeys.get(idx).getSecond(),
HConstants.EMPTY_BYTE_ARRAY)) {
+ throw new IOException("The last region info for table " + tableName
+ + " can't be found in hbase:meta.Please use hbck tool to fix it
first.");
+} else if (idx + 1 < startEndKeys.size() &&
!(Bytes.compareTo(startEndKeys.get(idx).getSecond(),
+startEndKeys.get(idx + 1).getFirst()) == 0)) {
+ throw new IOException("The endkey of one region for table " + tableName
+ + " is not equal to the startkey of the next region in
hbase:meta."
+ + "Please use hbck tool to fix it first.");
+}
+ }
+
+ /**
* Attempt to assign the given load queue item into its target region group.
If the hfile boundary
* no longer fits into a region, physically splits the hfile such that the
new bottom half will
* fit and returns the list of LQI's corresponding to the resultant hfiles.
@@ -647,51 +686,30 @@ public class BulkLoadHFilesTool extends Configured
implements BulkLoadHFiles, To
return null;
}
if (Bytes.compareTo(first.get(), last.get()) > 0) {
- throw new IllegalArgumentException("Invalid range: " +
Bytes.toStringBinary(first.get()) +
-" > " + Bytes.toStringBinary(last.get()));
-}
-int idx =
- Collections.binarySearch(startEndKeys, Pair.newPair(first.get(),
HConstants.EMPTY_END_ROW),
-(p1, p2) -> Bytes.compareTo(p1.getFirst(), p2.getFirst()));
-if (idx < 0) {
- // not on boundary, returns -(insertion index). Calculate region it
- // would be in.
- idx = -(idx + 1) - 1;
+ throw new IllegalArgumentException("Invalid range: " +
Bytes.toStringBinary(first.get())
+ + " > " + Bytes.toStringBinary(last.get()));
}
-int indexForCallable = idx;
-
-/*
- * we can consider there is a region hole in following conditions. 1) if
idx < 0,then first
- * region info is lost. 2) if the endkey of a region is not equal to the
startkey of the next
- * region. 3) if the endkey of the last region is not empty.
- */
-if (indexForCallable < 0) {
- throw new IOException("T
[hbase] branch branch-2 updated: HBASE-25213 Should request Compaction after bulkLoadHFiles is done (#2684)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-2 by this push:
new 8976781 HBASE-25213 Should request Compaction after bulkLoadHFiles is
done (#2684)
8976781 is described below
commit 897678130280a61e3d69900207b00561953911f3
Author: niuyulin
AuthorDate: Mon Nov 23 08:33:41 2020 +0800
HBASE-25213 Should request Compaction after bulkLoadHFiles is done (#2684)
Signed-off-by: Guanghao Zhang
---
.../apache/hadoop/hbase/regionserver/HRegion.java | 63 +++--
.../hadoop/hbase/regionserver/TestBulkLoad.java| 275 -
.../hbase/regionserver/TestBulkloadBase.java | 215
.../regionserver/TestCompactionAfterBulkLoad.java | 110 +
4 files changed, 414 insertions(+), 249 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
index c123faf..9645235 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -7047,6 +7047,19 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver, Regi
}
isSuccessful = true;
+ //request compaction
+ familyWithFinalPath.keySet().forEach(family -> {
+HStore store = getStore(family);
+try {
+ if (this.rsServices != null && store.needsCompaction()) {
+this.rsServices.getCompactionRequestor().requestCompaction(this,
store,
+ "bulkload hfiles request compaction", Store.PRIORITY_USER + 1,
+ CompactionLifeCycleTracker.DUMMY, null);
+ }
+} catch (IOException e) {
+ LOG.error("bulkload hfiles request compaction error ", e);
+}
+ });
} finally {
if (wal != null && !storeFiles.isEmpty()) {
// Write a bulk load event for hfiles that are loaded
@@ -7786,20 +7799,19 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver, Regi
// Utility methods
/**
- * A utility method to create new instances of HRegion based on the
- * {@link HConstants#REGION_IMPL} configuration property.
- * @param tableDir qualified path of directory where region should be
located,
- * usually the table directory.
- * @param wal The WAL is the outbound log for any updates to the HRegion
- * The wal file is a logfile from the previous execution that's
- * custom-computed for this HRegion. The HRegionServer computes and sorts the
- * appropriate wal info for this HRegion. If there is a previous file
- * (implying that the HRegion has been written-to before), then read it from
- * the supplied path.
+ * A utility method to create new instances of HRegion based on the {@link
HConstants#REGION_IMPL}
+ * configuration property.
+ * @param tableDir qualified path of directory where region should be
located, usually the table
+ * directory.
+ * @param wal The WAL is the outbound log for any updates to the HRegion The
wal file is a logfile
+ * from the previous execution that's custom-computed for this
HRegion. The HRegionServer
+ * computes and sorts the appropriate wal info for this HRegion. If
there is a previous
+ * file (implying that the HRegion has been written-to before),
then read it from the
+ * supplied path.
* @param fs is the filesystem.
* @param conf is global configuration settings.
- * @param regionInfo - RegionInfo that describes the region
- * is new), then read them from the supplied path.
+ * @param regionInfo - RegionInfo that describes the region is new), then
read them from the
+ * supplied path.
* @param htd the table descriptor
* @return the new instance
*/
@@ -7825,7 +7837,6 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver, Regi
/**
* Convenience method creating new HRegions. Used by createTable.
- *
* @param info Info for region to create.
* @param rootDir Root directory for HBase instance
* @param wal shared WAL
@@ -7833,14 +7844,30 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver, Regi
* @return new HRegion
*/
public static HRegion createHRegion(final RegionInfo info, final Path
rootDir,
-final Configuration conf, final TableDescriptor hTableDescriptor, final
WAL wal,
-final boolean initialize) throws IOException {
-LOG.info("creating " + info + ", tableDescriptor=" +
- (hTableDescriptor == null ? "null" : hTableDescriptor) + ", regionDir="
+ rootDir);
+ final Configuration conf
[hbase] branch master updated: HBASE-25213 Should request Compaction after bulkLoadHFiles is done (#2587)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/master by this push:
new 55399a0 HBASE-25213 Should request Compaction after bulkLoadHFiles is
done (#2587)
55399a0 is described below
commit 55399a03204cf46069808fdd87b3573db02a4015
Author: niuyulin
AuthorDate: Fri Nov 20 14:18:42 2020 +0800
HBASE-25213 Should request Compaction after bulkLoadHFiles is done (#2587)
Signed-off-by: Guanghao Zhang
Signed-off-by: Duo Zhang
---
.../apache/hadoop/hbase/regionserver/HRegion.java | 63 --
.../hadoop/hbase/regionserver/TestBulkLoad.java| 243 +++--
.../hbase/regionserver/TestBulkloadBase.java | 215 ++
.../regionserver/TestCompactionAfterBulkLoad.java | 110 ++
4 files changed, 398 insertions(+), 233 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
index 67764b9..200e02c 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -7096,6 +7096,19 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver, Regi
}
isSuccessful = true;
+ //request compaction
+ familyWithFinalPath.keySet().forEach(family -> {
+HStore store = getStore(family);
+try {
+ if (this.rsServices != null && store.needsCompaction()) {
+this.rsServices.getCompactionRequestor().requestCompaction(this,
store,
+ "bulkload hfiles request compaction", Store.PRIORITY_USER + 1,
+ CompactionLifeCycleTracker.DUMMY, null);
+ }
+} catch (IOException e) {
+ LOG.error("bulkload hfiles request compaction error ", e);
+}
+ });
} finally {
if (wal != null && !storeFiles.isEmpty()) {
// Write a bulk load event for hfiles that are loaded
@@ -7835,20 +7848,19 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver, Regi
// Utility methods
/**
- * A utility method to create new instances of HRegion based on the
- * {@link HConstants#REGION_IMPL} configuration property.
- * @param tableDir qualified path of directory where region should be
located,
- * usually the table directory.
- * @param wal The WAL is the outbound log for any updates to the HRegion
- * The wal file is a logfile from the previous execution that's
- * custom-computed for this HRegion. The HRegionServer computes and sorts the
- * appropriate wal info for this HRegion. If there is a previous file
- * (implying that the HRegion has been written-to before), then read it from
- * the supplied path.
+ * A utility method to create new instances of HRegion based on the {@link
HConstants#REGION_IMPL}
+ * configuration property.
+ * @param tableDir qualified path of directory where region should be
located, usually the table
+ * directory.
+ * @param wal The WAL is the outbound log for any updates to the HRegion The
wal file is a logfile
+ * from the previous execution that's custom-computed for this
HRegion. The HRegionServer
+ * computes and sorts the appropriate wal info for this HRegion. If
there is a previous
+ * file (implying that the HRegion has been written-to before),
then read it from the
+ * supplied path.
* @param fs is the filesystem.
* @param conf is global configuration settings.
- * @param regionInfo - RegionInfo that describes the region
- * is new), then read them from the supplied path.
+ * @param regionInfo - RegionInfo that describes the region is new), then
read them from the
+ * supplied path.
* @param htd the table descriptor
* @return the new instance
*/
@@ -7874,7 +7886,6 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver, Regi
/**
* Convenience method creating new HRegions. Used by createTable.
- *
* @param info Info for region to create.
* @param rootDir Root directory for HBase instance
* @param wal shared WAL
@@ -7882,14 +7893,30 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver, Regi
* @return new HRegion
*/
public static HRegion createHRegion(final RegionInfo info, final Path
rootDir,
-final Configuration conf, final TableDescriptor hTableDescriptor, final
WAL wal,
-final boolean initialize) throws IOException {
-LOG.info("creating " + info + ", tableDescriptor=" +
- (hTableDescriptor == null ? "null" : hTableDescriptor) + ", regionDir="
+ rootDir)
[hbase] branch branch-2.2 updated: HBASE-25300 'Unknown table hbase:quota' happens when desc table in shell if quota disabled (#2673)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a commit to branch branch-2.2 in repository https://gitbox.apache.org/repos/asf/hbase.git The following commit(s) were added to refs/heads/branch-2.2 by this push: new 3516cc3 HBASE-25300 'Unknown table hbase:quota' happens when desc table in shell if quota disabled (#2673) 3516cc3 is described below commit 3516cc3be0931eb165c05b5bb113708f99f0ccdf Author: XinSun AuthorDate: Thu Nov 19 09:16:25 2020 +0800 HBASE-25300 'Unknown table hbase:quota' happens when desc table in shell if quota disabled (#2673) Signed-off-by: Guanghao Zhang Signed-off-by: Pankaj Kumar --- hbase-shell/src/main/ruby/shell/commands/describe.rb | 17 - 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/hbase-shell/src/main/ruby/shell/commands/describe.rb b/hbase-shell/src/main/ruby/shell/commands/describe.rb index 5ef02a0..8d0b8c4 100644 --- a/hbase-shell/src/main/ruby/shell/commands/describe.rb +++ b/hbase-shell/src/main/ruby/shell/commands/describe.rb @@ -44,12 +44,19 @@ EOF puts end formatter.footer -puts -formatter.header(%w[QUOTAS]) -count = quotas_admin.list_quotas(TABLE => table.to_s) do |_, quota| - formatter.row([quota]) +if admin.exists?(::HBaseQuotasConstants::QUOTA_TABLE_NAME.to_s) + if table.to_s != 'hbase:meta' +# No QUOTAS if hbase:meta table +puts +formatter.header(%w[QUOTAS]) +count = quotas_admin.list_quotas(TABLE => table.to_s) do |_, quota| + formatter.row([quota]) +end +formatter.footer(count) + end +else + puts 'Quota is disabled' end -formatter.footer(count) end # rubocop:enable Metrics/AbcSize, Metrics/MethodLength end
[hbase] branch branch-2.3 updated: HBASE-25300 'Unknown table hbase:quota' happens when desc table in shell if quota disabled (#2673)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a commit to branch branch-2.3 in repository https://gitbox.apache.org/repos/asf/hbase.git The following commit(s) were added to refs/heads/branch-2.3 by this push: new e25f0fb HBASE-25300 'Unknown table hbase:quota' happens when desc table in shell if quota disabled (#2673) e25f0fb is described below commit e25f0fb6541fac8833818994d5483a85a8543039 Author: XinSun AuthorDate: Thu Nov 19 09:16:25 2020 +0800 HBASE-25300 'Unknown table hbase:quota' happens when desc table in shell if quota disabled (#2673) Signed-off-by: Guanghao Zhang Signed-off-by: Pankaj Kumar --- hbase-shell/src/main/ruby/shell/commands/describe.rb | 18 +++--- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/hbase-shell/src/main/ruby/shell/commands/describe.rb b/hbase-shell/src/main/ruby/shell/commands/describe.rb index 0553755..8d0b8c4 100644 --- a/hbase-shell/src/main/ruby/shell/commands/describe.rb +++ b/hbase-shell/src/main/ruby/shell/commands/describe.rb @@ -44,14 +44,18 @@ EOF puts end formatter.footer -if table.to_s != 'hbase:meta' - # No QUOTAS if hbase:meta table - puts - formatter.header(%w[QUOTAS]) - count = quotas_admin.list_quotas(TABLE => table.to_s) do |_, quota| -formatter.row([quota]) +if admin.exists?(::HBaseQuotasConstants::QUOTA_TABLE_NAME.to_s) + if table.to_s != 'hbase:meta' +# No QUOTAS if hbase:meta table +puts +formatter.header(%w[QUOTAS]) +count = quotas_admin.list_quotas(TABLE => table.to_s) do |_, quota| + formatter.row([quota]) +end +formatter.footer(count) end - formatter.footer(count) +else + puts 'Quota is disabled' end end # rubocop:enable Metrics/AbcSize, Metrics/MethodLength
[hbase] 01/02: HBASE-25300 'Unknown table hbase:quota' happens when desc table in shell if quota disabled (#2673)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a commit to branch branch-2 in repository https://gitbox.apache.org/repos/asf/hbase.git commit 9769e3fe7bdba9804b0b77e330f7e3d669553491 Author: XinSun AuthorDate: Thu Nov 19 09:16:25 2020 +0800 HBASE-25300 'Unknown table hbase:quota' happens when desc table in shell if quota disabled (#2673) Signed-off-by: Guanghao Zhang Signed-off-by: Pankaj Kumar --- hbase-shell/src/main/ruby/shell/commands/describe.rb | 18 +++--- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/hbase-shell/src/main/ruby/shell/commands/describe.rb b/hbase-shell/src/main/ruby/shell/commands/describe.rb index c32b77d..9865924 100644 --- a/hbase-shell/src/main/ruby/shell/commands/describe.rb +++ b/hbase-shell/src/main/ruby/shell/commands/describe.rb @@ -44,14 +44,18 @@ EOF puts end formatter.footer -if table.to_s != 'hbase:meta' - # No QUOTAS if hbase:meta table - puts - formatter.header(%w[QUOTAS]) - count = quotas_admin.list_quotas(::HBaseConstants::TABLE => table.to_s) do |_, quota| -formatter.row([quota]) +if admin.exists?(::HBaseQuotasConstants::QUOTA_TABLE_NAME.to_s) + if table.to_s != 'hbase:meta' +# No QUOTAS if hbase:meta table +puts +formatter.header(%w[QUOTAS]) +count = quotas_admin.list_quotas(::HBaseConstants::TABLE => table.to_s) do |_, quota| + formatter.row([quota]) +end +formatter.footer(count) end - formatter.footer(count) +else + puts 'Quota is disabled' end end # rubocop:enable Metrics/AbcSize, Metrics/MethodLength
[hbase] branch branch-2 updated (4d41de1 -> fdff8ef)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch branch-2 in repository https://gitbox.apache.org/repos/asf/hbase.git. from 4d41de1 HBASE-25083 further HBase 1.y releases should have Hadoop 2.10 as a minimum version. (#2656) new 9769e3f HBASE-25300 'Unknown table hbase:quota' happens when desc table in shell if quota disabled (#2673) new fdff8ef HBASE-25298 hbase.rsgroup.fallback.enable should support dynamic configuration (#2668) The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java | 12 ++-- .../master/balancer/TestRSGroupBasedLoadBalancer.java | 18 ++ hbase-shell/src/main/ruby/shell/commands/describe.rb | 18 +++--- src/main/asciidoc/_chapters/configuration.adoc | 1 + 4 files changed, 40 insertions(+), 9 deletions(-)
[hbase] 02/02: HBASE-25298 hbase.rsgroup.fallback.enable should support dynamic configuration (#2668)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit fdff8ef46dc458d43b5dfee93eb3b4c4fae680cf
Author: Baiqiang Zhao
AuthorDate: Thu Nov 19 09:20:01 2020 +0800
HBASE-25298 hbase.rsgroup.fallback.enable should support dynamic
configuration (#2668)
Signed-off-by: Guanghao Zhang
Signed-off-by: Viraj Jasani
---
.../hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java | 12 ++--
.../master/balancer/TestRSGroupBasedLoadBalancer.java | 18 ++
src/main/asciidoc/_chapters/configuration.adoc | 1 +
3 files changed, 29 insertions(+), 2 deletions(-)
diff --git
a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
b/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
index 92f8452..52f1681 100644
---
a/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
+++
b/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
@@ -247,7 +247,7 @@ public class RSGroupBasedLoadBalancer implements
RSGroupableBalancer {
}
if (!fallbackRegions.isEmpty()) {
List candidates = null;
-if (fallbackEnabled) {
+if (isFallbackEnabled()) {
candidates = getFallBackCandidates(servers);
}
candidates = (candidates == null || candidates.isEmpty()) ?
@@ -383,6 +383,9 @@ public class RSGroupBasedLoadBalancer implements
RSGroupableBalancer {
return this.rsGroupInfoManager.isOnline();
}
+ public boolean isFallbackEnabled() {
+return fallbackEnabled;
+ }
@Override
public void regionOnline(RegionInfo regionInfo, ServerName sn) {
@@ -394,7 +397,12 @@ public class RSGroupBasedLoadBalancer implements
RSGroupableBalancer {
@Override
public void onConfigurationChange(Configuration conf) {
-//DO nothing for now
+boolean newFallbackEnabled = conf.getBoolean(FALLBACK_GROUP_ENABLE_KEY,
false);
+if (fallbackEnabled != newFallbackEnabled) {
+ LOG.info("Changing the value of {} from {} to {}",
FALLBACK_GROUP_ENABLE_KEY,
+fallbackEnabled, newFallbackEnabled);
+ fallbackEnabled = newFallbackEnabled;
+}
}
@Override
diff --git
a/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java
b/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java
index 9b0202e..e7ee7a9 100644
---
a/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java
+++
b/hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java
@@ -18,6 +18,7 @@
package org.apache.hadoop.hbase.master.balancer;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;
import java.util.ArrayList;
@@ -186,4 +187,21 @@ public class TestRSGroupBasedLoadBalancer extends
RSGroupableBalancerTestBase {
.roundRobinAssignment(regions, onlineServers);
assertEquals(bogusRegion,
assignments.get(LoadBalancer.BOGUS_SERVER_NAME).size());
}
+
+ @Test
+ public void testOnConfigurationChange() {
+// fallbackEnabled default is false
+assertFalse(loadBalancer.isFallbackEnabled());
+
+// change FALLBACK_GROUP_ENABLE_KEY from false to true
+Configuration conf = loadBalancer.getConf();
+conf.setBoolean(RSGroupBasedLoadBalancer.FALLBACK_GROUP_ENABLE_KEY, true);
+loadBalancer.onConfigurationChange(conf);
+assertTrue(loadBalancer.isFallbackEnabled());
+
+// restore
+conf.setBoolean(RSGroupBasedLoadBalancer.FALLBACK_GROUP_ENABLE_KEY, false);
+loadBalancer.onConfigurationChange(conf);
+assertFalse(loadBalancer.isFallbackEnabled());
+ }
}
diff --git a/src/main/asciidoc/_chapters/configuration.adoc
b/src/main/asciidoc/_chapters/configuration.adoc
index b25ee96..5e5ecd5 100644
--- a/src/main/asciidoc/_chapters/configuration.adoc
+++ b/src/main/asciidoc/_chapters/configuration.adoc
@@ -1383,6 +1383,7 @@ Here are those configurations:
| hbase.master.balancer.stochastic.tableSkewCost
| hbase.master.regions.recovery.check.interval
| hbase.regions.recovery.store.file.ref.count
+| hbase.rsgroup.fallback.enable
|===
ifdef::backend-docbook[]
[hbase] branch master updated: HBASE-25298 hbase.rsgroup.fallback.enable should support dynamic configuration (#2668)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/master by this push:
new 8c1e476 HBASE-25298 hbase.rsgroup.fallback.enable should support
dynamic configuration (#2668)
8c1e476 is described below
commit 8c1e4763b3e11d4553e5a59e620ab30e3b2047e9
Author: Baiqiang Zhao
AuthorDate: Thu Nov 19 09:20:01 2020 +0800
HBASE-25298 hbase.rsgroup.fallback.enable should support dynamic
configuration (#2668)
Signed-off-by: Guanghao Zhang
Signed-off-by: Viraj Jasani
---
.../hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java | 12 ++--
.../master/balancer/TestRSGroupBasedLoadBalancer.java | 18 ++
src/main/asciidoc/_chapters/configuration.adoc | 1 +
3 files changed, 29 insertions(+), 2 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
index db61c01..27d407e 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
@@ -250,7 +250,7 @@ public class RSGroupBasedLoadBalancer implements
LoadBalancer {
}
if (!fallbackRegions.isEmpty()) {
List candidates = null;
-if (fallbackEnabled) {
+if (isFallbackEnabled()) {
candidates = getFallBackCandidates(servers);
}
candidates = (candidates == null || candidates.isEmpty()) ?
@@ -385,6 +385,9 @@ public class RSGroupBasedLoadBalancer implements
LoadBalancer {
return this.rsGroupInfoManager.isOnline();
}
+ public boolean isFallbackEnabled() {
+return fallbackEnabled;
+ }
@Override
public void regionOnline(RegionInfo regionInfo, ServerName sn) {
@@ -396,7 +399,12 @@ public class RSGroupBasedLoadBalancer implements
LoadBalancer {
@Override
public void onConfigurationChange(Configuration conf) {
-//DO nothing for now
+boolean newFallbackEnabled = conf.getBoolean(FALLBACK_GROUP_ENABLE_KEY,
false);
+if (fallbackEnabled != newFallbackEnabled) {
+ LOG.info("Changing the value of {} from {} to {}",
FALLBACK_GROUP_ENABLE_KEY,
+fallbackEnabled, newFallbackEnabled);
+ fallbackEnabled = newFallbackEnabled;
+}
}
@Override
diff --git
a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java
b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java
index 15a8d63..0da0e14 100644
---
a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java
+++
b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRSGroupBasedLoadBalancer.java
@@ -18,6 +18,7 @@
package org.apache.hadoop.hbase.master.balancer;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;
import java.util.ArrayList;
@@ -185,4 +186,21 @@ public class TestRSGroupBasedLoadBalancer extends
RSGroupableBalancerTestBase {
loadBalancer.roundRobinAssignment(regions, onlineServers);
assertEquals(bogusRegion,
assignments.get(LoadBalancer.BOGUS_SERVER_NAME).size());
}
+
+ @Test
+ public void testOnConfigurationChange() {
+// fallbackEnabled default is false
+assertFalse(loadBalancer.isFallbackEnabled());
+
+// change FALLBACK_GROUP_ENABLE_KEY from false to true
+Configuration conf = loadBalancer.getConf();
+conf.setBoolean(RSGroupBasedLoadBalancer.FALLBACK_GROUP_ENABLE_KEY, true);
+loadBalancer.onConfigurationChange(conf);
+assertTrue(loadBalancer.isFallbackEnabled());
+
+// restore
+conf.setBoolean(RSGroupBasedLoadBalancer.FALLBACK_GROUP_ENABLE_KEY, false);
+loadBalancer.onConfigurationChange(conf);
+assertFalse(loadBalancer.isFallbackEnabled());
+ }
}
diff --git a/src/main/asciidoc/_chapters/configuration.adoc
b/src/main/asciidoc/_chapters/configuration.adoc
index 93e905a..0b2f5da 100644
--- a/src/main/asciidoc/_chapters/configuration.adoc
+++ b/src/main/asciidoc/_chapters/configuration.adoc
@@ -1355,6 +1355,7 @@ Here are those configurations:
| hbase.master.balancer.stochastic.tableSkewCost
| hbase.master.regions.recovery.check.interval
| hbase.regions.recovery.store.file.ref.count
+| hbase.rsgroup.fallback.enable
|===
ifdef::backend-docbook[]
[hbase] branch master updated: HBASE-25300 'Unknown table hbase:quota' happens when desc table in shell if quota disabled (#2673)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hbase.git The following commit(s) were added to refs/heads/master by this push: new 2b61b99 HBASE-25300 'Unknown table hbase:quota' happens when desc table in shell if quota disabled (#2673) 2b61b99 is described below commit 2b61b99668bfc7b969a60c88aaf020b984c881a9 Author: XinSun AuthorDate: Thu Nov 19 09:16:25 2020 +0800 HBASE-25300 'Unknown table hbase:quota' happens when desc table in shell if quota disabled (#2673) Signed-off-by: Guanghao Zhang Signed-off-by: Pankaj Kumar --- hbase-shell/src/main/ruby/shell/commands/describe.rb | 18 +++--- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/hbase-shell/src/main/ruby/shell/commands/describe.rb b/hbase-shell/src/main/ruby/shell/commands/describe.rb index c32b77d..9865924 100644 --- a/hbase-shell/src/main/ruby/shell/commands/describe.rb +++ b/hbase-shell/src/main/ruby/shell/commands/describe.rb @@ -44,14 +44,18 @@ EOF puts end formatter.footer -if table.to_s != 'hbase:meta' - # No QUOTAS if hbase:meta table - puts - formatter.header(%w[QUOTAS]) - count = quotas_admin.list_quotas(::HBaseConstants::TABLE => table.to_s) do |_, quota| -formatter.row([quota]) +if admin.exists?(::HBaseQuotasConstants::QUOTA_TABLE_NAME.to_s) + if table.to_s != 'hbase:meta' +# No QUOTAS if hbase:meta table +puts +formatter.header(%w[QUOTAS]) +count = quotas_admin.list_quotas(::HBaseConstants::TABLE => table.to_s) do |_, quota| + formatter.row([quota]) +end +formatter.footer(count) end - formatter.footer(count) +else + puts 'Quota is disabled' end end # rubocop:enable Metrics/AbcSize, Metrics/MethodLength
[hbase] branch branch-2 updated (e8ecf2e -> d9b175c)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch branch-2 in repository https://gitbox.apache.org/repos/asf/hbase.git. from e8ecf2e HBASE-25261 Upgrade Bootstrap to 3.4.1 add d9b175c HBASE-25289 [testing] Clean up resources after tests in rsgroup_shell_test.rb (#2672) No new revisions were added by this update. Summary of changes: .../src/test/ruby/shell/rsgroup_shell_test.rb | 61 +- 1 file changed, 49 insertions(+), 12 deletions(-)
[hbase] branch branch-2.3 updated: HBASE-25289 [testing] Clean up resources after tests in rsgroup_shell_test.rb (#2672)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch branch-2.3
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-2.3 by this push:
new 927c109 HBASE-25289 [testing] Clean up resources after tests in
rsgroup_shell_test.rb (#2672)
927c109 is described below
commit 927c109e4c4d26a6ce4bac1eadfe2ed4fcc6214f
Author: XinSun
AuthorDate: Wed Nov 18 14:29:02 2020 +0800
HBASE-25289 [testing] Clean up resources after tests in
rsgroup_shell_test.rb (#2672)
Signed-off-by: Guanghao Zhang
---
.../src/test/ruby/shell/rsgroup_shell_test.rb | 49 +++---
1 file changed, 44 insertions(+), 5 deletions(-)
diff --git a/hbase-shell/src/test/ruby/shell/rsgroup_shell_test.rb
b/hbase-shell/src/test/ruby/shell/rsgroup_shell_test.rb
index 5e7f37d..a5fd0ca 100644
--- a/hbase-shell/src/test/ruby/shell/rsgroup_shell_test.rb
+++ b/hbase-shell/src/test/ruby/shell/rsgroup_shell_test.rb
@@ -30,6 +30,23 @@ module Hbase
org.apache.hadoop.hbase.rsgroup.RSGroupAdminClient.new(connection)
end
+def add_rsgroup_and_move_one_server(group_name)
+ assert_nil(@rsgroup_admin.getRSGroupInfo(group_name))
+ @shell.command(:add_rsgroup, group_name)
+ assert_not_nil(@rsgroup_admin.getRSGroupInfo(group_name))
+
+ hostport =
@rsgroup_admin.getRSGroupInfo('default').getServers.iterator.next
+ @shell.command(:move_servers_rsgroup, group_name, [hostport.toString])
+ assert_equal(1,
@rsgroup_admin.getRSGroupInfo(group_name).getServers.count)
+end
+
+def remove_rsgroup(group_name)
+ rsgroup = @rsgroup_admin.getRSGroupInfo(group_name)
+ @rsgroup_admin.moveServers(rsgroup.getServers, 'default')
+ @rsgroup_admin.removeRSGroup(group_name)
+ assert_nil(@rsgroup_admin.getRSGroupInfo(group_name))
+end
+
define_test 'Test Basic RSGroup Commands' do
group_name = 'test_group'
table_name = 'test_table'
@@ -73,6 +90,10 @@ module Hbase
# just run it to verify jruby->java api binding
@hbase.rsgroup_admin.balance_rs_group(group_name)
+
+ @shell.command(:disable, table_name)
+ @shell.command(:drop, table_name)
+ remove_rsgroup(group_name)
end
define_test 'Test RSGroup Move Namespace RSGroup Commands' do
@@ -80,17 +101,24 @@ module Hbase
namespace_name = 'test_namespace'
ns_table_name = 'test_namespace:test_ns_table'
+ add_rsgroup_and_move_one_server(group_name)
+
@shell.command('create_namespace', namespace_name)
@shell.command('create', ns_table_name, 'f')
@shell.command('move_namespaces_rsgroup',
group_name,
[namespace_name])
- assert_equal(2,
@rsgroup_admin.getRSGroupInfo(group_name).getTables.count)
+ assert_equal(1,
@rsgroup_admin.getRSGroupInfo(group_name).getTables.count)
group = @hbase.rsgroup_admin.get_rsgroup(group_name)
assert_not_nil(group)
assert_equal(ns_table_name, group.getTables.iterator.next.toString)
+
+ @shell.command(:disable, ns_table_name)
+ @shell.command(:drop, ns_table_name)
+ @shell.command(:drop_namespace, namespace_name)
+ remove_rsgroup(group_name)
end
define_test 'Test RSGroup Move Server Namespace RSGroup Commands' do
@@ -98,13 +126,11 @@ module Hbase
namespace_name = 'test_namespace'
ns_table_name = 'test_namespace:test_ns_table'
+ @shell.command('create_namespace', namespace_name)
+ @shell.command('create', ns_table_name, 'f')
@shell.command('add_rsgroup', ns_group_name)
assert_not_nil(@rsgroup_admin.getRSGroupInfo(ns_group_name))
- @shell.command('move_tables_rsgroup',
- 'default',
- [ns_table_name])
-
group_servers = @rsgroup_admin.getRSGroupInfo('default').getServers
hostport_str = group_servers.iterator.next.toString
@shell.command('move_servers_namespaces_rsgroup',
@@ -115,6 +141,11 @@ module Hbase
assert_not_nil(ns_group)
assert_equal(hostport_str, ns_group.getServers.iterator.next.toString)
assert_equal(ns_table_name, ns_group.getTables.iterator.next.toString)
+
+ @shell.command(:disable, ns_table_name)
+ @shell.command(:drop, ns_table_name)
+ @shell.command(:drop_namespace, namespace_name)
+ remove_rsgroup(ns_group_name)
end
# we test exceptions that could be thrown by the ruby wrappers
@@ -135,12 +166,20 @@ module Hbase
new_rs_group_name = 'renamed_test_group'
table_name = 'test_table'
+ add_rsgroup_and_move_one_server(old_rs_group_name)
+
[hbase] branch branch-2 updated (e8ecf2e -> d9b175c)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch branch-2 in repository https://gitbox.apache.org/repos/asf/hbase.git. from e8ecf2e HBASE-25261 Upgrade Bootstrap to 3.4.1 add d9b175c HBASE-25289 [testing] Clean up resources after tests in rsgroup_shell_test.rb (#2672) No new revisions were added by this update. Summary of changes: .../src/test/ruby/shell/rsgroup_shell_test.rb | 61 +- 1 file changed, 49 insertions(+), 12 deletions(-)
[hbase] branch master updated: HBASE-25289 [testing] Clean up resources after tests in rsgroup_shell_test.rb (#2659)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/master by this push:
new 9419c78 HBASE-25289 [testing] Clean up resources after tests in
rsgroup_shell_test.rb (#2659)
9419c78 is described below
commit 9419c78d26fea25b1e6e38929acb3f0dabe31cab
Author: XinSun
AuthorDate: Wed Nov 18 09:32:06 2020 +0800
HBASE-25289 [testing] Clean up resources after tests in
rsgroup_shell_test.rb (#2659)
Signed-off-by: Guanghao Zhang
---
.../src/test/ruby/shell/rsgroup_shell_test.rb | 65 +-
1 file changed, 51 insertions(+), 14 deletions(-)
diff --git a/hbase-shell/src/test/ruby/shell/rsgroup_shell_test.rb
b/hbase-shell/src/test/ruby/shell/rsgroup_shell_test.rb
index e8ba851..32980b8 100644
--- a/hbase-shell/src/test/ruby/shell/rsgroup_shell_test.rb
+++ b/hbase-shell/src/test/ruby/shell/rsgroup_shell_test.rb
@@ -29,6 +29,23 @@ module Hbase
@admin = connection.getAdmin
end
+def add_rsgroup_and_move_one_server(group_name)
+ assert_nil(@admin.getRSGroup(group_name))
+ @shell.command(:add_rsgroup, group_name)
+ assert_not_nil(@admin.getRSGroup(group_name))
+
+ hostport = @admin.getRSGroup('default').getServers.iterator.next
+ @shell.command(:move_servers_rsgroup, group_name, [hostport.toString])
+ assert_equal(1, @admin.getRSGroup(group_name).getServers.count)
+end
+
+def remove_rsgroup(group_name)
+ rsgroup = @admin.getRSGroup(group_name)
+ @admin.moveServersToRSGroup(rsgroup.getServers, 'default')
+ @admin.removeRSGroup(group_name)
+ assert_nil(@admin.getRSGroup(group_name))
+end
+
define_test 'Test Basic RSGroup Commands' do
group_name = 'test_group'
table_name = 'test_table'
@@ -71,6 +88,10 @@ module Hbase
# just run it to verify jruby->java api binding
@hbase.rsgroup_admin.balance_rs_group(group_name)
+
+ @shell.command(:disable, table_name)
+ @shell.command(:drop, table_name)
+ remove_rsgroup(group_name)
end
define_test 'Test RSGroup Move Namespace RSGroup Commands' do
@@ -78,17 +99,24 @@ module Hbase
namespace_name = 'test_namespace'
ns_table_name = 'test_namespace:test_ns_table'
- @shell.command('create_namespace', namespace_name)
- @shell.command('create', ns_table_name, 'f')
+ add_rsgroup_and_move_one_server(group_name)
+
+ @shell.command(:create_namespace, namespace_name)
+ @shell.command(:create, ns_table_name, 'f')
@shell.command('move_namespaces_rsgroup',
group_name,
[namespace_name])
- assert_equal(2, @admin.listTablesInRSGroup(group_name).count)
+ assert_equal(1, @admin.listTablesInRSGroup(group_name).count)
group = @hbase.rsgroup_admin.get_rsgroup(group_name)
assert_not_nil(group)
assert_true(@admin.listTablesInRSGroup(group_name).contains(org.apache.hadoop.hbase.TableName.valueOf(ns_table_name)))
+
+ @shell.command(:disable, ns_table_name)
+ @shell.command(:drop, ns_table_name)
+ @shell.command(:drop_namespace, namespace_name)
+ remove_rsgroup(group_name)
end
define_test 'Test RSGroup Move Server Namespace RSGroup Commands' do
@@ -96,13 +124,11 @@ module Hbase
namespace_name = 'test_namespace'
ns_table_name = 'test_namespace:test_ns_table'
+ @shell.command('create_namespace', namespace_name)
+ @shell.command('create', ns_table_name, 'f')
@shell.command('add_rsgroup', ns_group_name)
assert_not_nil(@admin.getRSGroup(ns_group_name))
- @shell.command('move_tables_rsgroup',
- 'default',
- [ns_table_name])
-
group_servers = @admin.getRSGroup('default').getServers
hostport_str = group_servers.iterator.next.toString
@shell.command('move_servers_namespaces_rsgroup',
@@ -113,6 +139,11 @@ module Hbase
assert_not_nil(ns_group)
assert_equal(hostport_str, ns_group.getServers.iterator.next.toString)
assert_equal(ns_table_name,
@admin.listTablesInRSGroup(ns_group_name).iterator.next.toString)
+
+ @shell.command(:disable, ns_table_name)
+ @shell.command(:drop, ns_table_name)
+ @shell.command(:drop_namespace, namespace_name)
+ remove_rsgroup(ns_group_name)
end
# we test exceptions that could be thrown by the ruby wrappers
@@ -133,26 +164,32 @@ module Hbase
new_rs_group_name = 'renamed_test_group'
table_name = 'test_table'
+ add_rsgroup_and_move_one_server(old_rs_group_name)
+
[hbase] branch master updated: HBASE-25289 [testing] Clean up resources after tests in rsgroup_shell_test.rb (#2659)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/master by this push:
new 9419c78 HBASE-25289 [testing] Clean up resources after tests in
rsgroup_shell_test.rb (#2659)
9419c78 is described below
commit 9419c78d26fea25b1e6e38929acb3f0dabe31cab
Author: XinSun
AuthorDate: Wed Nov 18 09:32:06 2020 +0800
HBASE-25289 [testing] Clean up resources after tests in
rsgroup_shell_test.rb (#2659)
Signed-off-by: Guanghao Zhang
---
.../src/test/ruby/shell/rsgroup_shell_test.rb | 65 +-
1 file changed, 51 insertions(+), 14 deletions(-)
diff --git a/hbase-shell/src/test/ruby/shell/rsgroup_shell_test.rb
b/hbase-shell/src/test/ruby/shell/rsgroup_shell_test.rb
index e8ba851..32980b8 100644
--- a/hbase-shell/src/test/ruby/shell/rsgroup_shell_test.rb
+++ b/hbase-shell/src/test/ruby/shell/rsgroup_shell_test.rb
@@ -29,6 +29,23 @@ module Hbase
@admin = connection.getAdmin
end
+def add_rsgroup_and_move_one_server(group_name)
+ assert_nil(@admin.getRSGroup(group_name))
+ @shell.command(:add_rsgroup, group_name)
+ assert_not_nil(@admin.getRSGroup(group_name))
+
+ hostport = @admin.getRSGroup('default').getServers.iterator.next
+ @shell.command(:move_servers_rsgroup, group_name, [hostport.toString])
+ assert_equal(1, @admin.getRSGroup(group_name).getServers.count)
+end
+
+def remove_rsgroup(group_name)
+ rsgroup = @admin.getRSGroup(group_name)
+ @admin.moveServersToRSGroup(rsgroup.getServers, 'default')
+ @admin.removeRSGroup(group_name)
+ assert_nil(@admin.getRSGroup(group_name))
+end
+
define_test 'Test Basic RSGroup Commands' do
group_name = 'test_group'
table_name = 'test_table'
@@ -71,6 +88,10 @@ module Hbase
# just run it to verify jruby->java api binding
@hbase.rsgroup_admin.balance_rs_group(group_name)
+
+ @shell.command(:disable, table_name)
+ @shell.command(:drop, table_name)
+ remove_rsgroup(group_name)
end
define_test 'Test RSGroup Move Namespace RSGroup Commands' do
@@ -78,17 +99,24 @@ module Hbase
namespace_name = 'test_namespace'
ns_table_name = 'test_namespace:test_ns_table'
- @shell.command('create_namespace', namespace_name)
- @shell.command('create', ns_table_name, 'f')
+ add_rsgroup_and_move_one_server(group_name)
+
+ @shell.command(:create_namespace, namespace_name)
+ @shell.command(:create, ns_table_name, 'f')
@shell.command('move_namespaces_rsgroup',
group_name,
[namespace_name])
- assert_equal(2, @admin.listTablesInRSGroup(group_name).count)
+ assert_equal(1, @admin.listTablesInRSGroup(group_name).count)
group = @hbase.rsgroup_admin.get_rsgroup(group_name)
assert_not_nil(group)
assert_true(@admin.listTablesInRSGroup(group_name).contains(org.apache.hadoop.hbase.TableName.valueOf(ns_table_name)))
+
+ @shell.command(:disable, ns_table_name)
+ @shell.command(:drop, ns_table_name)
+ @shell.command(:drop_namespace, namespace_name)
+ remove_rsgroup(group_name)
end
define_test 'Test RSGroup Move Server Namespace RSGroup Commands' do
@@ -96,13 +124,11 @@ module Hbase
namespace_name = 'test_namespace'
ns_table_name = 'test_namespace:test_ns_table'
+ @shell.command('create_namespace', namespace_name)
+ @shell.command('create', ns_table_name, 'f')
@shell.command('add_rsgroup', ns_group_name)
assert_not_nil(@admin.getRSGroup(ns_group_name))
- @shell.command('move_tables_rsgroup',
- 'default',
- [ns_table_name])
-
group_servers = @admin.getRSGroup('default').getServers
hostport_str = group_servers.iterator.next.toString
@shell.command('move_servers_namespaces_rsgroup',
@@ -113,6 +139,11 @@ module Hbase
assert_not_nil(ns_group)
assert_equal(hostport_str, ns_group.getServers.iterator.next.toString)
assert_equal(ns_table_name,
@admin.listTablesInRSGroup(ns_group_name).iterator.next.toString)
+
+ @shell.command(:disable, ns_table_name)
+ @shell.command(:drop, ns_table_name)
+ @shell.command(:drop_namespace, namespace_name)
+ remove_rsgroup(ns_group_name)
end
# we test exceptions that could be thrown by the ruby wrappers
@@ -133,26 +164,32 @@ module Hbase
new_rs_group_name = 'renamed_test_group'
table_name = 'test_table'
+ add_rsgroup_and_move_one_server(old_rs_group_name)
+
[hbase] branch master updated (300b0a6 -> 322435d)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hbase.git. from 300b0a6 HBASE-25026 Create a metric to track full region scans RPCs add 322435d HBASE-25296 [Documentation] fix duplicate conf entry about upgrading (#2666) No new revisions were added by this update. Summary of changes: src/main/asciidoc/_chapters/upgrading.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[hbase] branch branch-2.2 updated: HBASE-25276 Need to throw the original exception in HRegion#openHRegion (#2648)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch branch-2.2
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-2.2 by this push:
new af70af3 HBASE-25276 Need to throw the original exception in
HRegion#openHRegion (#2648)
af70af3 is described below
commit af70af3b3150a8096501e9369dacd22fe7382c04
Author: niuyulin
AuthorDate: Thu Nov 12 17:40:24 2020 +0800
HBASE-25276 Need to throw the original exception in HRegion#openHRegion
(#2648)
Signed-off-by: Guanghao Zhang
---
.../main/java/org/apache/hadoop/hbase/regionserver/HRegion.java | 9 +++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
index 97306d4..1232d49 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -7374,11 +7374,16 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver, Regi
RegionReplicaUtil.isDefaultReplica(getRegionInfo())) {
writeRegionOpenMarker(wal, openSeqNum);
}
-}catch(Throwable t){
+} catch (Throwable t) {
// By coprocessor path wrong region will open failed,
// MetricsRegionWrapperImpl is already init and not close,
// add region close when open failed
- this.close();
+ try {
+this.close();
+ } catch (Throwable e) {
+LOG.warn("Open region: {} failed. Try close region but got exception
", this.getRegionInfo(),
+ e);
+ }
throw t;
}
return this;
[hbase] branch branch-2.3 updated: HBASE-25276 Need to throw the original exception in HRegion#openHRegion (#2648)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch branch-2.3
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-2.3 by this push:
new 0c1f9ab HBASE-25276 Need to throw the original exception in
HRegion#openHRegion (#2648)
0c1f9ab is described below
commit 0c1f9abe7182b8b83f0ffd266223503988766428
Author: niuyulin
AuthorDate: Thu Nov 12 17:40:24 2020 +0800
HBASE-25276 Need to throw the original exception in HRegion#openHRegion
(#2648)
Signed-off-by: Guanghao Zhang
---
.../main/java/org/apache/hadoop/hbase/regionserver/HRegion.java | 9 +++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
index f450dab..df04b3a 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -7506,11 +7506,16 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver, Regi
RegionReplicaUtil.isDefaultReplica(getRegionInfo())) {
writeRegionOpenMarker(wal, openSeqNum);
}
-}catch(Throwable t){
+} catch (Throwable t) {
// By coprocessor path wrong region will open failed,
// MetricsRegionWrapperImpl is already init and not close,
// add region close when open failed
- this.close();
+ try {
+this.close();
+ } catch (Throwable e) {
+LOG.warn("Open region: {} failed. Try close region but got exception
", this.getRegionInfo(),
+ e);
+ }
throw t;
}
return this;
[hbase] branch branch-2 updated: HBASE-25276 Need to throw the original exception in HRegion#openHRegion (#2648)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-2 by this push:
new c48f6eb HBASE-25276 Need to throw the original exception in
HRegion#openHRegion (#2648)
c48f6eb is described below
commit c48f6ebd09fd7c09268499e13011322212d1d78f
Author: niuyulin
AuthorDate: Thu Nov 12 17:40:24 2020 +0800
HBASE-25276 Need to throw the original exception in HRegion#openHRegion
(#2648)
Signed-off-by: Guanghao Zhang
---
.../main/java/org/apache/hadoop/hbase/regionserver/HRegion.java | 9 +++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
index 3a08239..a7a06f9 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -8085,11 +8085,16 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver, Regi
RegionReplicaUtil.isDefaultReplica(getRegionInfo())) {
writeRegionOpenMarker(wal, openSeqNum);
}
-}catch(Throwable t){
+} catch (Throwable t) {
// By coprocessor path wrong region will open failed,
// MetricsRegionWrapperImpl is already init and not close,
// add region close when open failed
- this.close();
+ try {
+this.close();
+ } catch (Throwable e) {
+LOG.warn("Open region: {} failed. Try close region but got exception
", this.getRegionInfo(),
+ e);
+ }
throw t;
}
return this;
[hbase] branch master updated: HBASE-25276 Need to throw the original exception in HRegion#openHRegion (#2648)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/master by this push:
new 0b6d6fd HBASE-25276 Need to throw the original exception in
HRegion#openHRegion (#2648)
0b6d6fd is described below
commit 0b6d6fd773cf75dc49151975ed0b4ee82c741f59
Author: niuyulin
AuthorDate: Thu Nov 12 17:40:24 2020 +0800
HBASE-25276 Need to throw the original exception in HRegion#openHRegion
(#2648)
Signed-off-by: Guanghao Zhang
---
.../main/java/org/apache/hadoop/hbase/regionserver/HRegion.java | 9 +++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
index d0e6284..67764b9 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -8137,11 +8137,16 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver, Regi
RegionReplicaUtil.isDefaultReplica(getRegionInfo())) {
writeRegionOpenMarker(wal, openSeqNum);
}
-} catch(Throwable t) {
+} catch (Throwable t) {
// By coprocessor path wrong region will open failed,
// MetricsRegionWrapperImpl is already init and not close,
// add region close when open failed
- this.close();
+ try {
+this.close();
+ } catch (Throwable e) {
+LOG.warn("Open region: {} failed. Try close region but got exception
", this.getRegionInfo(),
+ e);
+ }
throw t;
}
return this;
[hbase] 02/09: HBASE-24681 Remove the cache walsById/walsByIdRecoveredQueues from ReplicationSourceManager (#2019)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch HBASE-24666
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit efbb75c19883742c824a4a914f70299e529d68a8
Author: Guanghao Zhang
AuthorDate: Mon Jul 13 17:35:32 2020 +0800
HBASE-24681 Remove the cache walsById/walsByIdRecoveredQueues from
ReplicationSourceManager (#2019)
Signed-off-by: Wellington Chevreuil
---
.../regionserver/ReplicationSourceManager.java | 204 +++--
1 file changed, 62 insertions(+), 142 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
index 3869857..a222f4b 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
@@ -91,30 +91,6 @@ import
org.apache.hbase.thirdparty.com.google.common.util.concurrent.ThreadFacto
* No need synchronized on {@link #sources}. {@link #sources} is a
ConcurrentHashMap and there
* is a Lock for peer id in {@link PeerProcedureHandlerImpl}. So there is no
race for peer
* operations.
- * Need synchronized on {@link #walsById}. There are four methods which
modify it,
- * {@link #addPeer(String)}, {@link #removePeer(String)},
- * {@link #cleanOldLogs(String, boolean, ReplicationSourceInterface)} and
{@link #preLogRoll(Path)}.
- * {@link #walsById} is a ConcurrentHashMap and there is a Lock for peer id in
- * {@link PeerProcedureHandlerImpl}. So there is no race between {@link
#addPeer(String)} and
- * {@link #removePeer(String)}. {@link #cleanOldLogs(String, boolean,
ReplicationSourceInterface)}
- * is called by {@link ReplicationSourceInterface}. So no race with {@link
#addPeer(String)}.
- * {@link #removePeer(String)} will terminate the {@link
ReplicationSourceInterface} firstly, then
- * remove the wals from {@link #walsById}. So no race with {@link
#removePeer(String)}. The only
- * case need synchronized is {@link #cleanOldLogs(String, boolean,
ReplicationSourceInterface)} and
- * {@link #preLogRoll(Path)}.
- * No need synchronized on {@link #walsByIdRecoveredQueues}. There are
three methods which
- * modify it, {@link #removePeer(String)} ,
- * {@link #cleanOldLogs(String, boolean, ReplicationSourceInterface)} and
- * {@link ReplicationSourceManager.NodeFailoverWorker#run()}.
- * {@link #cleanOldLogs(String, boolean, ReplicationSourceInterface)} is
called by
- * {@link ReplicationSourceInterface}. {@link #removePeer(String)} will
terminate the
- * {@link ReplicationSourceInterface} firstly, then remove the wals from
- * {@link #walsByIdRecoveredQueues}. And {@link
ReplicationSourceManager.NodeFailoverWorker#run()}
- * will add the wals to {@link #walsByIdRecoveredQueues} firstly, then start
up a
- * {@link ReplicationSourceInterface}. So there is no race here. For
- * {@link ReplicationSourceManager.NodeFailoverWorker#run()} and {@link
#removePeer(String)}, there
- * is already synchronized on {@link #oldsources}. So no need synchronized on
- * {@link #walsByIdRecoveredQueues}.
* Need synchronized on {@link #latestPaths} to avoid the new open source
miss new log.
* Need synchronized on {@link #oldsources} to avoid adding recovered
source for the
* to-be-removed peer.
@@ -135,15 +111,6 @@ public class ReplicationSourceManager implements
ReplicationListener {
// All about stopping
private final Server server;
- // All logs we are currently tracking
- // Index structure of the map is: queue_id->logPrefix/logGroup->logs
- // For normal replication source, the peer id is same with the queue id
- private final ConcurrentMap>>
walsById;
- // Logs for recovered sources we are currently tracking
- // the map is: queue_id->logPrefix/logGroup->logs
- // For recovered source, the queue id's format is peer_id-servername-*
- private final ConcurrentMap>>
walsByIdRecoveredQueues;
-
private final SyncReplicationPeerMappingManager
syncReplicationPeerMappingManager;
private final Configuration conf;
@@ -199,8 +166,6 @@ public class ReplicationSourceManager implements
ReplicationListener {
this.replicationPeers = replicationPeers;
this.replicationTracker = replicationTracker;
this.server = server;
-this.walsById = new ConcurrentHashMap<>();
-this.walsByIdRecoveredQueues = new ConcurrentHashMap<>();
this.oldsources = new ArrayList<>();
this.conf = conf;
this.fs = fs;
@@ -338,7 +303,6 @@ public class ReplicationSourceManager implements
ReplicationListener {
// Delete queue from storage and memory and queue id is same with peer
id for normal
// source
deleteQueue(peerId);
- this.walsById.re
[hbase] branch HBASE-24666 updated (3640775 -> 34e49bc)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch HBASE-24666 in repository https://gitbox.apache.org/repos/asf/hbase.git. discard 3640775 HBASE-25071 ReplicationServer support start ReplicationSource internal (#2452) omit f67c3df HBASE-24999 Master manages ReplicationServers (#2579) omit 1a64f1a HBASE-24684 Fetch ReplicationSink servers list from HMaster instead o… (#2077) omit 4f77158 HBASE-24998 Introduce a ReplicationSourceOverallController interface and decouple ReplicationSourceManager and ReplicationSource (#2364) omit 8ed76d4 HBASE-24982 Disassemble the method replicateWALEntry from AdminService to a new interface ReplicationServerService (#2360) omit d43d2c9 HBASE-24683 Add a basic ReplicationServer which only implement ReplicationSink Service (#2111) omit 43410f9 HBASE-24735: Refactor ReplicationSourceManager: move logPositionAndCleanOldLogs/cleanUpHFileRefs to ReplicationSource inside (#2064) omit 25764a3 HBASE-24681 Remove the cache walsById/walsByIdRecoveredQueues from ReplicationSourceManager (#2019) omit 7a41a3e HBASE-24682 Refactor ReplicationSource#addHFileRefs method: move it to ReplicationSourceManager (#2020) add 74df1e1 HBASE-25207 Revisit the implementation and usage of RegionStates.include (#2571) add d8de24c HBASE-25128 RSGroupInfo's toString() and hashCode() does not take into account configuration map. (#2484) add 5e3ffb1 HBASE-25193: Add support for row prefix and type in the WAL Pretty Printer add 881c92b HBASE-25206 Data loss can happen if a cloned table loses original split region(delete table) (#2569) add e5d4e2f HBASE-25189 [Metrics] Add checkAndPut and checkAndDelete latency metrics at table level (#2549) add 164cc5a HBASE-25215 TestClientOperationTimeout.testScanTimeout is flaky (#2583) add 4bbc772 HBASE-25173 Remove owner related methods in TableDescriptor/TableDescriptorBuilder (#2541) add 485da75 HBASE-25176 MasterStoppedException should be moved to hbase-client module (#2538) add 17f9ade HBASE-25201 YouAreDeadException should be moved to hbase-server module (#2581) add 735689d HBASE-25223 Use try-with-resources statement (#2592) add bb4a9d3 HBASE-25224 Maximize sleep for checking meta and namespace regions availability (#2593) add 259fe19 HBASE-24845 Git/Jira Release Audit: limit branches when building audit db (#2238) add 35b344c HBASE-25090 CompactionConfiguration logs unrealistic store file sizes (#2595) add 12d0397 HBASE-24419 Normalizer merge plans should consider more than 2 regions when possible add eee1cf7 HBASE-24200 Upgrade to Yetus 0.12.0 add e3beccf HBASE-24977 Meta table shouldn't be modified as read only (#2537) add d790bde HBASE-25167 Normalizer support for hot config reloading (#2523) add 85dfd9a HBASE-25218 : Add 2.3.3 to the downloads page add 7a41247 HBASE-25228 Delete dev-support/jenkins_precommit_jira_yetus.sh (#2611) add 5abbda1 HBASE-24859: Optimize in-memory representation of HBase map reduce table splits (#2609) add 1eceab6 HBASE-25210 RegionInfo.isOffline is now a duplication with RegionInfo.isSplit (#2580) add c98e993 HBASE-25212 Optionally abort requests in progress after deciding a region should close (#2574) add f37cd05 HBASE-25235 Cleanup the deprecated methods in TimeRange (#2616) add 49774c7 HBASE-25216 The client zk syncer should deal with meta replica count change (#2614) add 4bd9ee4 HBASE-25245 : Fixing incorrect maven and jdk names for generate-hbase-website add 0e71d61 HBASE-25053 WAL replay should ignore 0-length files (#2437) add 0356e8e HBASE-25240 gson format of RpcServer.logResponse is abnormal add 23e6567 HBASE-25238 Upgrading HBase from 2.2.0 to 2.3.x fails because of “Message missing required fields: state” (#2625) add 671129d HBASE-25252 Move HMaster inner classes out (#2628) add c36ee4f HBASE-25254 Rewrite TestMultiLogThreshold to remove the LogDelegate in RSRpcServices (#2631) new ee34412 HBASE-24682 Refactor ReplicationSource#addHFileRefs method: move it to ReplicationSourceManager (#2020) new efbb75c HBASE-24681 Remove the cache walsById/walsByIdRecoveredQueues from ReplicationSourceManager (#2019) new a86c174 HBASE-24735: Refactor ReplicationSourceManager: move logPositionAndCleanOldLogs/cleanUpHFileRefs to ReplicationSource inside (#2064) new 1bcf389 HBASE-24683 Add a basic ReplicationServer which only implement ReplicationSink Service (#2111) new 83dcae9 HBASE-24982 Disassemble the method replicateWALEntry from AdminService to a new interface ReplicationServerService (#2360) new 13006e4 HBASE-24998 Introduce a ReplicationSourceOverallController interface and decouple ReplicationSourceManager and ReplicationSource (#2364) new 3ef10b0 HBASE-24684 Fetch Repli
[hbase] 01/09: HBASE-24682 Refactor ReplicationSource#addHFileRefs method: move it to ReplicationSourceManager (#2020)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch HBASE-24666
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit ee344123006ebfab950e14a78595d476aeb8d03d
Author: Guanghao Zhang
AuthorDate: Wed Jul 8 14:29:08 2020 +0800
HBASE-24682 Refactor ReplicationSource#addHFileRefs method: move it to
ReplicationSourceManager (#2020)
Signed-off-by: Wellington Chevreuil
---
.../regionserver/ReplicationSource.java| 38 +
.../regionserver/ReplicationSourceInterface.java | 14 ---
.../regionserver/ReplicationSourceManager.java | 48 +-
.../hbase/replication/ReplicationSourceDummy.java | 9 +---
4 files changed, 49 insertions(+), 60 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
index 8091d0c..a49cd5d 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
@@ -27,7 +27,6 @@ import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
-import java.util.Set;
import java.util.TreeMap;
import java.util.UUID;
import java.util.concurrent.ConcurrentHashMap;
@@ -37,6 +36,7 @@ import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicLong;
import java.util.function.Predicate;
+
import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
@@ -46,21 +46,17 @@ import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.Server;
import org.apache.hadoop.hbase.ServerName;
import org.apache.hadoop.hbase.TableDescriptors;
-import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.regionserver.HRegionServer;
import org.apache.hadoop.hbase.regionserver.RSRpcServices;
import org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost;
import org.apache.hadoop.hbase.replication.ChainWALEntryFilter;
import org.apache.hadoop.hbase.replication.ClusterMarkingEntryFilter;
import org.apache.hadoop.hbase.replication.ReplicationEndpoint;
-import org.apache.hadoop.hbase.replication.ReplicationException;
import org.apache.hadoop.hbase.replication.ReplicationPeer;
import org.apache.hadoop.hbase.replication.ReplicationQueueInfo;
import org.apache.hadoop.hbase.replication.ReplicationQueueStorage;
import org.apache.hadoop.hbase.replication.SystemTableWALEntryFilter;
import org.apache.hadoop.hbase.replication.WALEntryFilter;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.hadoop.hbase.util.Pair;
import org.apache.hadoop.hbase.util.Threads;
import org.apache.hadoop.hbase.wal.AbstractFSWALProvider;
import org.apache.hadoop.hbase.wal.WAL.Entry;
@@ -281,38 +277,6 @@ public class ReplicationSource implements
ReplicationSourceInterface {
}
}
- @Override
- public void addHFileRefs(TableName tableName, byte[] family, List> pairs)
- throws ReplicationException {
-String peerId = replicationPeer.getId();
-Set namespaces = replicationPeer.getNamespaces();
-Map> tableCFMap = replicationPeer.getTableCFs();
-if (tableCFMap != null) { // All peers with TableCFs
- List tableCfs = tableCFMap.get(tableName);
- if (tableCFMap.containsKey(tableName)
- && (tableCfs == null || tableCfs.contains(Bytes.toString(family {
-this.queueStorage.addHFileRefs(peerId, pairs);
-metrics.incrSizeOfHFileRefsQueue(pairs.size());
- } else {
-LOG.debug("HFiles will not be replicated belonging to the table {}
family {} to peer id {}",
-tableName, Bytes.toString(family), peerId);
- }
-} else if (namespaces != null) { // Only for set NAMESPACES peers
- if (namespaces.contains(tableName.getNamespaceAsString())) {
-this.queueStorage.addHFileRefs(peerId, pairs);
-metrics.incrSizeOfHFileRefsQueue(pairs.size());
- } else {
-LOG.debug("HFiles will not be replicated belonging to the table {}
family {} to peer id {}",
-tableName, Bytes.toString(family), peerId);
- }
-} else {
- // user has explicitly not defined any table cfs for replication, means
replicate all the
- // data
- this.queueStorage.addHFileRefs(peerId, pairs);
- metrics.incrSizeOfHFileRefsQueue(pairs.size());
-}
- }
-
private ReplicationEndpoint createReplicationEndpoint()
throws InstantiationException, IllegalAccessException,
ClassNotFoundException, IOException {
RegionServerCoprocessorHost rsServerHost = null;
diff --git
a/hbase-server/src/main/java/o
[hbase] 06/09: HBASE-24998 Introduce a ReplicationSourceOverallController interface and decouple ReplicationSourceManager and ReplicationSource (#2364)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch HBASE-24666
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 13006e47c9f59a497048efc36d9521cd8970
Author: Guanghao Zhang
AuthorDate: Sun Sep 20 09:02:53 2020 +0800
HBASE-24998 Introduce a ReplicationSourceOverallController interface and
decouple ReplicationSourceManager and ReplicationSource (#2364)
Signed-off-by: meiyi
---
.../java/org/apache/hadoop/hbase/HConstants.java | 2 +
.../hbase/replication/ReplicationListener.java | 2 +-
.../replication/ReplicationSourceController.java | 31 +++--
.../regionserver/RecoveredReplicationSource.java | 18 ++-
.../regionserver/ReplicationSource.java| 31 ++---
.../regionserver/ReplicationSourceInterface.java | 25 ++--
.../regionserver/ReplicationSourceManager.java | 141 +++--
.../regionserver/ReplicationSourceWALReader.java | 13 +-
.../hbase/replication/ReplicationSourceDummy.java | 21 ++-
.../regionserver/TestReplicationSourceManager.java | 11 +-
.../regionserver/TestWALEntryStream.java | 15 ++-
11 files changed, 167 insertions(+), 143 deletions(-)
diff --git a/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
b/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
index e1d3de9..5ffd1d6 100644
--- a/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
+++ b/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
@@ -979,6 +979,8 @@ public final class HConstants {
/*
* cluster replication constants.
*/
+ public static final String REPLICATION_OFFLOAD_ENABLE_KEY =
"hbase.replication.offload.enabled";
+ public static final boolean REPLICATION_OFFLOAD_ENABLE_DEFAULT = false;
public static final String
REPLICATION_SOURCE_SERVICE_CLASSNAME =
"hbase.replication.source.service";
public static final String REPLICATION_SERVICE_CLASSNAME_DEFAULT =
diff --git
a/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ReplicationListener.java
b/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ReplicationListener.java
index f040bf9..6ecbb46 100644
---
a/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ReplicationListener.java
+++
b/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ReplicationListener.java
@@ -33,5 +33,5 @@ public interface ReplicationListener {
* A region server has been removed from the local cluster
* @param regionServer the removed region server
*/
- public void regionServerRemoved(String regionServer);
+ void regionServerRemoved(String regionServer);
}
diff --git
a/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ReplicationListener.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ReplicationSourceController.java
similarity index 50%
copy from
hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ReplicationListener.java
copy to
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ReplicationSourceController.java
index f040bf9..5bb9dd6 100644
---
a/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ReplicationListener.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ReplicationSourceController.java
@@ -1,5 +1,4 @@
-/*
- *
+/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
@@ -18,20 +17,32 @@
*/
package org.apache.hadoop.hbase.replication;
+import java.util.concurrent.atomic.AtomicLong;
+
+import
org.apache.hadoop.hbase.replication.regionserver.MetricsReplicationGlobalSourceSource;
+import
org.apache.hadoop.hbase.replication.regionserver.RecoveredReplicationSource;
import org.apache.yetus.audience.InterfaceAudience;
/**
- * The replication listener interface can be implemented if a class needs to
subscribe to events
- * generated by the ReplicationTracker. These events include things like
addition/deletion of peer
- * clusters or failure of a local region server. To receive events, the class
also needs to register
- * itself with a Replication Tracker.
+ * Used to control all replication sources inside one RegionServer or
ReplicationServer.
+ * Used by {@link
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource} or
+ * {@link RecoveredReplicationSource}.
*/
@InterfaceAudience.Private
-public interface ReplicationListener {
+public interface ReplicationSourceController {
+
+ /**
+ * Returns the maximum size in bytes of edits held in memory which are
pending replication
+ * across all sources inside this RegionServer or ReplicationServer.
+ */
+ long getTotalBufferLimit();
+
+ AtomicLong getTotalBufferUsed();
+
+ MetricsReplicationGlobalSourceSource getGlobalMetrics
[hbase] 05/09: HBASE-24982 Disassemble the method replicateWALEntry from AdminService to a new interface ReplicationServerService (#2360)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch HBASE-24666
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 83dcae9921253f31cd7731e1f1421a817d968bc4
Author: XinSun
AuthorDate: Wed Sep 9 15:00:37 2020 +0800
HBASE-24982 Disassemble the method replicateWALEntry from AdminService to a
new interface ReplicationServerService (#2360)
Signed-off-by: Wellington Chevreuil
---
.../hadoop/hbase/client/AsyncConnectionImpl.java | 16 ++
.../server/replication/ReplicationServer.proto | 32
.../hadoop/hbase/replication/ReplicationUtils.java | 19 ++
.../hbase/client/AsyncClusterConnection.java | 5 +
.../hbase/client/AsyncClusterConnectionImpl.java | 5 +
.../hbase/client/AsyncReplicationServerAdmin.java | 80 +
.../hbase/protobuf/ReplicationProtobufUtil.java| 18 ++
.../hadoop/hbase/regionserver/RSRpcServices.java | 8 +-
.../replication/HBaseReplicationEndpoint.java | 57 +-
.../replication/ReplicationServerRpcServices.java | 200 +
.../HBaseInterClusterReplicationEndpoint.java | 7 +-
.../regionserver/ReplicationSource.java| 2 +-
.../hbase/client/DummyAsyncClusterConnection.java | 5 +
.../replication/TestHBaseReplicationEndpoint.java | 17 +-
.../hbase/replication/TestReplicationServer.java | 43 -
.../regionserver/TestReplicationSource.java| 2 +-
16 files changed, 290 insertions(+), 226 deletions(-)
diff --git
a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncConnectionImpl.java
b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncConnectionImpl.java
index 406af0d..e8f0fb0 100644
---
a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncConnectionImpl.java
+++
b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncConnectionImpl.java
@@ -66,6 +66,7 @@ import
org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos.AdminServic
import
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos.ClientService;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos;
import
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.MasterService;
+import
org.apache.hadoop.hbase.shaded.protobuf.generated.ReplicationServerProtos.ReplicationServerService;
/**
* The implementation of AsyncConnection.
@@ -107,6 +108,8 @@ class AsyncConnectionImpl implements AsyncConnection {
private final ConcurrentMap rsStubs = new
ConcurrentHashMap<>();
private final ConcurrentMap adminSubs = new
ConcurrentHashMap<>();
+ private final ConcurrentMap
replStubs =
+ new ConcurrentHashMap<>();
private final AtomicReference masterStub = new
AtomicReference<>();
@@ -266,12 +269,25 @@ class AsyncConnectionImpl implements AsyncConnection {
return AdminService.newStub(rpcClient.createRpcChannel(serverName, user,
rpcTimeout));
}
+ private ReplicationServerService.Interface
createReplicationServerStub(ServerName serverName)
+ throws IOException {
+return ReplicationServerService.newStub(
+rpcClient.createRpcChannel(serverName, user, rpcTimeout));
+ }
+
AdminService.Interface getAdminStub(ServerName serverName) throws
IOException {
return ConcurrentMapUtils.computeIfAbsentEx(adminSubs,
getStubKey(AdminService.getDescriptor().getName(), serverName,
hostnameCanChange),
() -> createAdminServerStub(serverName));
}
+ ReplicationServerService.Interface getReplicationServerStub(ServerName
serverName)
+ throws IOException {
+return ConcurrentMapUtils.computeIfAbsentEx(replStubs,
+getStubKey(ReplicationServerService.Interface.class.getSimpleName(),
serverName,
+hostnameCanChange), () -> createReplicationServerStub(serverName));
+ }
+
CompletableFuture getMasterStub() {
return ConnectionUtils.getOrFetch(masterStub, masterStubMakeFuture, false,
() -> {
CompletableFuture future = new
CompletableFuture<>();
diff --git
a/hbase-protocol-shaded/src/main/protobuf/server/replication/ReplicationServer.proto
b/hbase-protocol-shaded/src/main/protobuf/server/replication/ReplicationServer.proto
new file mode 100644
index 000..ed334c4
--- /dev/null
+++
b/hbase-protocol-shaded/src/main/protobuf/server/replication/ReplicationServer.proto
@@ -0,0 +1,32 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed
[hbase] 04/09: HBASE-24683 Add a basic ReplicationServer which only implement ReplicationSink Service (#2111)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch HBASE-24666
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 1bcf3890c2f78c15ab17a343e4df593eb1abea17
Author: XinSun
AuthorDate: Fri Sep 4 18:53:46 2020 +0800
HBASE-24683 Add a basic ReplicationServer which only implement
ReplicationSink Service (#2111)
Signed-off-by: Guanghao Zhang
---
.../java/org/apache/hadoop/hbase/util/DNS.java | 3 +-
.../hbase/replication/HReplicationServer.java | 391
.../replication/ReplicationServerRpcServices.java | 516 +
.../hbase/replication/TestReplicationServer.java | 151 ++
4 files changed, 1060 insertions(+), 1 deletion(-)
diff --git a/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DNS.java
b/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DNS.java
index 2b4e1cb..ddff6db 100644
--- a/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DNS.java
+++ b/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DNS.java
@@ -54,7 +54,8 @@ public final class DNS {
public enum ServerType {
MASTER("master"),
-REGIONSERVER("regionserver");
+REGIONSERVER("regionserver"),
+REPLICATIONSERVER("replicationserver");
private String name;
ServerType(String name) {
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HReplicationServer.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HReplicationServer.java
new file mode 100644
index 000..31dec0c
--- /dev/null
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HReplicationServer.java
@@ -0,0 +1,391 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.replication;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.ChoreService;
+import org.apache.hadoop.hbase.CoordinatedStateManager;
+import org.apache.hadoop.hbase.DoNotRetryIOException;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.Server;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.ClusterConnectionFactory;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.log.HBaseMarkers;
+import org.apache.hadoop.hbase.regionserver.ReplicationService;
+import org.apache.hadoop.hbase.regionserver.ReplicationSinkService;
+import org.apache.hadoop.hbase.security.User;
+import org.apache.hadoop.hbase.security.UserProvider;
+import org.apache.hadoop.hbase.trace.TraceUtil;
+import org.apache.hadoop.hbase.util.Sleeper;
+import org.apache.hadoop.hbase.zookeeper.ZKWatcher;
+import org.apache.hadoop.util.ReflectionUtils;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * HReplicationServer which is responsible to all replication stuff. It checks
in with
+ * the HMaster. There are many HReplicationServers in a single HBase
deployment.
+ */
+@InterfaceAudience.Private
+@SuppressWarnings({ "deprecation"})
+public class HReplicationServer extends Thread implements Server {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(HReplicationServer.class);
+
+ /** replication server process name */
+ public static final String REPLICATION_SERVER = "replicationserver";
+
+ /**
+ * This servers start code.
+ */
+ protected final long startCode;
+
+ private volatile boolean stopped = false;
+
+ // Go down hard. Used if file system becomes unavailable and also in
+ // debugging and unit tests.
+ private AtomicBoolean abortRequested;
+
+ // flag set after we're done setting up server threads
+ final AtomicBoolean online = new AtomicBoolean(false);
+
+ /**
+ * The server name the Master sees us as. Its made from the hostname the
+ * master passes us, port, and server start code. Gets set after registration
+ * against Mas
[hbase] 07/09: HBASE-24684 Fetch ReplicationSink servers list from HMaster instead o… (#2077)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch HBASE-24666
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 3ef10b0cf8924e1b085b4c022feca31d332169a1
Author: XinSun
AuthorDate: Sun Sep 20 10:54:43 2020 +0800
HBASE-24684 Fetch ReplicationSink servers list from HMaster instead o…
(#2077)
Signed-off-by: Wellington Chevreuil
---
.../src/main/protobuf/server/master/Master.proto | 12 +-
.../hadoop/hbase/coprocessor/MasterObserver.java | 16 +++
.../org/apache/hadoop/hbase/master/HMaster.java| 5 +
.../hadoop/hbase/master/MasterCoprocessorHost.java | 18 +++
.../hadoop/hbase/master/MasterRpcServices.java | 21 +++
.../apache/hadoop/hbase/master/MasterServices.java | 6 +
.../replication/HBaseReplicationEndpoint.java | 146 +++--
.../regionserver/ReplicationSource.java| 4 +-
.../hbase/master/MockNoopMasterServices.java | 5 +
.../replication/TestHBaseReplicationEndpoint.java | 5 +
.../replication/TestReplicationFetchServers.java | 106 +++
.../TestGlobalReplicationThrottler.java| 4 +
...stRegionReplicaReplicationEndpointNoMaster.java | 2 +
13 files changed, 334 insertions(+), 16 deletions(-)
diff --git a/hbase-protocol-shaded/src/main/protobuf/server/master/Master.proto
b/hbase-protocol-shaded/src/main/protobuf/server/master/Master.proto
index 118ce77..7dec566 100644
--- a/hbase-protocol-shaded/src/main/protobuf/server/master/Master.proto
+++ b/hbase-protocol-shaded/src/main/protobuf/server/master/Master.proto
@@ -717,6 +717,13 @@ message BalancerDecisionsResponse {
repeated BalancerDecision balancer_decision = 1;
}
+message ListReplicationSinkServersRequest {
+}
+
+message ListReplicationSinkServersResponse {
+ repeated ServerName server_name = 1;
+}
+
service MasterService {
/** Used by the client to get the number of regions that have received the
updated schema */
rpc GetSchemaAlterStatus(GetSchemaAlterStatusRequest)
@@ -1146,10 +1153,13 @@ service MasterService {
returns (RenameRSGroupResponse);
rpc UpdateRSGroupConfig(UpdateRSGroupConfigRequest)
- returns (UpdateRSGroupConfigResponse);
+returns (UpdateRSGroupConfigResponse);
rpc GetLogEntries(LogRequest)
returns(LogEntry);
+
+ rpc ListReplicationSinkServers(ListReplicationSinkServersRequest)
+returns (ListReplicationSinkServersResponse);
}
// HBCK Service definitions.
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
index ac35caa..ec009cc 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
@@ -1782,4 +1782,20 @@ public interface MasterObserver {
default void
postHasUserPermissions(ObserverContext ctx,
String userName, List permissions) throws IOException {
}
+
+ /**
+ * Called before getting servers for replication sink.
+ * @param ctx the coprocessor instance's environment
+ */
+ default void
preListReplicationSinkServers(ObserverContext ctx)
+throws IOException {
+ }
+
+ /**
+ * Called after getting servers for replication sink.
+ * @param ctx the coprocessor instance's environment
+ */
+ default void
postListReplicationSinkServers(ObserverContext
ctx)
+throws IOException {
+ }
}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
index 573838f..2048ff9 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
@@ -3759,4 +3759,9 @@ public class HMaster extends HRegionServer implements
MasterServices {
public MetaLocationSyncer getMetaLocationSyncer() {
return metaLocationSyncer;
}
+
+ @Override
+ public List listReplicationSinkServers() throws IOException {
+return this.serverManager.getOnlineServersList();
+ }
}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
index 01d1a62..f775eba 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
@@ -2038,4 +2038,22 @@ public class MasterCoprocessorHost
}
});
}
+
+ public void preListReplicationSinkServers() throws IOException {
+execOperation(coprocEnvironments.isEmpty() ? null : new
MasterObserverOperation() {
+ @Override
+ public void call(MasterObserver observer) throws IOException {
+observer.preListReplicationSinkSe
[hbase] 08/09: HBASE-24999 Master manages ReplicationServers (#2579)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch HBASE-24666
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit e118d8dd9b4933a0eccc720010e0175afda78ae9
Author: XinSun
AuthorDate: Wed Oct 28 18:59:57 2020 +0800
HBASE-24999 Master manages ReplicationServers (#2579)
Signed-off-by: Guanghao Zhang
---
.../server/master/ReplicationServerStatus.proto| 34
.../org/apache/hadoop/hbase/master/HMaster.java| 10 +
.../hadoop/hbase/master/MasterRpcServices.java | 37 +++-
.../apache/hadoop/hbase/master/MasterServices.java | 5 +
.../hbase/master/ReplicationServerManager.java | 204
.../replication/HBaseReplicationEndpoint.java | 150 +++
.../hbase/replication/HReplicationServer.java | 214 -
.../HBaseInterClusterReplicationEndpoint.java | 1 -
.../regionserver/ReplicationSyncUp.java| 4 +-
.../hbase/master/MockNoopMasterServices.java | 5 +
.../hbase/replication/TestReplicationBase.java | 2 +
.../hbase/replication/TestReplicationServer.java | 57 +-
12 files changed, 620 insertions(+), 103 deletions(-)
diff --git
a/hbase-protocol-shaded/src/main/protobuf/server/master/ReplicationServerStatus.proto
b/hbase-protocol-shaded/src/main/protobuf/server/master/ReplicationServerStatus.proto
new file mode 100644
index 000..d39a043
--- /dev/null
+++
b/hbase-protocol-shaded/src/main/protobuf/server/master/ReplicationServerStatus.proto
@@ -0,0 +1,34 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+syntax = "proto2";
+
+package hbase.pb;
+
+option java_package = "org.apache.hadoop.hbase.shaded.protobuf.generated";
+option java_outer_classname = "ReplicationServerStatusProtos";
+option java_generic_services = true;
+option java_generate_equals_and_hash = true;
+option optimize_for = SPEED;
+
+import "server/master/RegionServerStatus.proto";
+
+service ReplicationServerStatusService {
+
+ rpc ReplicationServerReport(RegionServerReportRequest)
+ returns(RegionServerReportResponse);
+}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
index 2048ff9..e575b6a 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
@@ -304,6 +304,8 @@ public class HMaster extends HRegionServer implements
MasterServices {
// manager of assignment nodes in zookeeper
private AssignmentManager assignmentManager;
+ // server manager to deal with replication server info
+ private ReplicationServerManager replicationServerManager;
/**
* Cache for the meta region replica's locations. Also tracks their changes
to avoid stale
@@ -867,6 +869,8 @@ public class HMaster extends HRegionServer implements
MasterServices {
.collect(Collectors.toList());
this.assignmentManager.setupRIT(ritList);
+this.replicationServerManager = new ReplicationServerManager(this);
+
// Start RegionServerTracker with listing of servers found with exiting
SCPs -- these should
// be registered in the deadServers set -- and with the list of
servernames out on the
// filesystem that COULD BE 'alive' (we'll schedule SCPs for each and let
SCP figure it out).
@@ -1035,6 +1039,7 @@ public class HMaster extends HRegionServer implements
MasterServices {
this.hbckChore = new HbckChore(this);
getChoreService().scheduleChore(hbckChore);
this.serverManager.startChore();
+this.replicationServerManager.startChore();
// Only for rolling upgrade, where we need to migrate the data in
namespace table to meta table.
if (!waitForNamespaceOnline()) {
@@ -1294,6 +1299,11 @@ public class HMaster extends HRegionServer implements
MasterServices {
}
@Override
+ public ReplicationServerManager getReplicationServerManager() {
+return this.replicationServerManager;
+ }
+
+ @Override
public MasterFileSystem getMasterFileSystem() {
[hbase] 03/09: HBASE-24735: Refactor ReplicationSourceManager: move logPositionAndCleanOldLogs/cleanUpHFileRefs to ReplicationSource inside (#2064)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch HBASE-24666
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit a86c174ce344205a284898ca92ed633009fc9a26
Author: Guanghao Zhang
AuthorDate: Tue Aug 11 20:07:09 2020 +0800
HBASE-24735: Refactor ReplicationSourceManager: move
logPositionAndCleanOldLogs/cleanUpHFileRefs to ReplicationSource inside (#2064)
Signed-off-by: Wellington Chevreuil
---
.../regionserver/RecoveredReplicationSource.java | 18 ++-
.../regionserver/ReplicationSource.java| 160 +++--
.../regionserver/ReplicationSourceInterface.java | 33 +++--
.../regionserver/ReplicationSourceManager.java | 158 ++--
.../regionserver/ReplicationSourceShipper.java | 17 +--
.../regionserver/ReplicationSourceWALReader.java | 17 +--
.../SerialReplicationSourceWALReader.java | 1 +
.../replication/regionserver/WALEntryBatch.java| 2 +-
.../hbase/replication/ReplicationSourceDummy.java | 24 ++--
.../regionserver/TestReplicationSource.java| 12 +-
.../regionserver/TestReplicationSourceManager.java | 50 +++
.../regionserver/TestWALEntryStream.java | 20 +--
12 files changed, 258 insertions(+), 254 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RecoveredReplicationSource.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RecoveredReplicationSource.java
index 46cf851..e3400ad 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RecoveredReplicationSource.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RecoveredReplicationSource.java
@@ -21,6 +21,7 @@ import java.io.IOException;
import java.util.List;
import java.util.UUID;
import java.util.concurrent.PriorityBlockingQueue;
+
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
@@ -44,15 +45,18 @@ public class RecoveredReplicationSource extends
ReplicationSource {
private static final Logger LOG =
LoggerFactory.getLogger(RecoveredReplicationSource.class);
+ private Path walDir;
+
private String actualPeerId;
@Override
- public void init(Configuration conf, FileSystem fs, ReplicationSourceManager
manager,
- ReplicationQueueStorage queueStorage, ReplicationPeer replicationPeer,
Server server,
- String peerClusterZnode, UUID clusterId, WALFileLengthProvider
walFileLengthProvider,
- MetricsSource metrics) throws IOException {
-super.init(conf, fs, manager, queueStorage, replicationPeer, server,
peerClusterZnode,
+ public void init(Configuration conf, FileSystem fs, Path walDir,
ReplicationSourceManager manager,
+ReplicationQueueStorage queueStorage, ReplicationPeer replicationPeer,
Server server,
+String peerClusterZnode, UUID clusterId, WALFileLengthProvider
walFileLengthProvider,
+MetricsSource metrics) throws IOException {
+super.init(conf, fs, walDir, manager, queueStorage, replicationPeer,
server, peerClusterZnode,
clusterId, walFileLengthProvider, metrics);
+this.walDir = walDir;
this.actualPeerId = this.replicationQueueInfo.getPeerId();
}
@@ -93,7 +97,7 @@ public class RecoveredReplicationSource extends
ReplicationSource {
deadRsDirectory.suffix(AbstractFSWALProvider.SPLITTING_EXT),
path.getName()) };
for (Path possibleLogLocation : locs) {
LOG.info("Possible location " +
possibleLogLocation.toUri().toString());
-if (manager.getFs().exists(possibleLogLocation)) {
+if (this.fs.exists(possibleLogLocation)) {
// We found the right new location
LOG.info("Log " + path + " still exists at " +
possibleLogLocation);
newPaths.add(possibleLogLocation);
@@ -126,7 +130,7 @@ public class RecoveredReplicationSource extends
ReplicationSource {
// N.B. the ReplicationSyncUp tool sets the manager.getWALDir to the root of
the wal
// area rather than to the wal area for a particular region server.
private Path getReplSyncUpPath(Path path) throws IOException {
-FileStatus[] rss = fs.listStatus(manager.getLogDir());
+FileStatus[] rss = fs.listStatus(walDir);
for (FileStatus rs : rss) {
Path p = rs.getPath();
FileStatus[] logs = fs.listStatus(p);
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
index a49cd5d..702a8a2 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource
[hbase] 09/09: HBASE-25071 ReplicationServer support start ReplicationSource internal (#2452)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch HBASE-24666
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 34e49bc8f659506874735effd2d015599ee5d890
Author: Guanghao Zhang
AuthorDate: Mon Nov 9 11:46:02 2020 +0800
HBASE-25071 ReplicationServer support start ReplicationSource internal
(#2452)
Signed-off-by: XinSun
---
.../server/replication/ReplicationServer.proto | 14 +-
.../replication/ZKReplicationQueueStorage.java | 4 +-
.../replication/ZKReplicationStorageBase.java | 4 +
.../hadoop/hbase/master/MasterRpcServices.java | 2 +-
.../hadoop/hbase/regionserver/RSRpcServices.java | 4 -
.../replication/HBaseReplicationEndpoint.java | 14 +-
.../hbase/replication/HReplicationServer.java | 175 ++---
.../replication/ReplicationServerRpcServices.java | 15 ++
.../regionserver/RecoveredReplicationSource.java | 9 +-
.../regionserver/ReplicationSource.java| 58 ++-
.../regionserver/ReplicationSourceFactory.java | 2 +-
.../regionserver/ReplicationSourceInterface.java | 6 +-
.../regionserver/ReplicationSourceManager.java | 4 +-
.../hbase/replication/ReplicationSourceDummy.java | 5 +-
.../hbase/replication/TestReplicationBase.java | 2 +-
.../replication/TestReplicationFetchServers.java | 43 +++--
...nServer.java => TestReplicationServerSink.java} | 24 +--
.../replication/TestReplicationServerSource.java | 69
.../regionserver/TestReplicationSource.java| 12 +-
.../regionserver/TestReplicationSourceManager.java | 16 +-
20 files changed, 394 insertions(+), 88 deletions(-)
diff --git
a/hbase-protocol-shaded/src/main/protobuf/server/replication/ReplicationServer.proto
b/hbase-protocol-shaded/src/main/protobuf/server/replication/ReplicationServer.proto
index ed334c4..925aed4 100644
---
a/hbase-protocol-shaded/src/main/protobuf/server/replication/ReplicationServer.proto
+++
b/hbase-protocol-shaded/src/main/protobuf/server/replication/ReplicationServer.proto
@@ -24,9 +24,21 @@ option java_generic_services = true;
option java_generate_equals_and_hash = true;
option optimize_for = SPEED;
+import "HBase.proto";
import "server/region/Admin.proto";
+message StartReplicationSourceRequest {
+ required ServerName server_name = 1;
+ required string queue_id = 2;
+}
+
+message StartReplicationSourceResponse {
+}
+
service ReplicationServerService {
rpc ReplicateWALEntry(ReplicateWALEntryRequest)
returns(ReplicateWALEntryResponse);
-}
\ No newline at end of file
+
+ rpc StartReplicationSource(StartReplicationSourceRequest)
+returns(StartReplicationSourceResponse);
+}
diff --git
a/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationQueueStorage.java
b/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationQueueStorage.java
index 6f1f5a3..2b9594e 100644
---
a/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationQueueStorage.java
+++
b/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationQueueStorage.java
@@ -80,7 +80,7 @@ import
org.apache.hbase.thirdparty.org.apache.commons.collections4.CollectionUti
*
*/
@InterfaceAudience.Private
-class ZKReplicationQueueStorage extends ZKReplicationStorageBase
+public class ZKReplicationQueueStorage extends ZKReplicationStorageBase
implements ReplicationQueueStorage {
private static final Logger LOG =
LoggerFactory.getLogger(ZKReplicationQueueStorage.class);
@@ -123,7 +123,7 @@ class ZKReplicationQueueStorage extends
ZKReplicationStorageBase
return ZNodePaths.joinZNode(queuesZNode, serverName.getServerName());
}
- private String getQueueNode(ServerName serverName, String queueId) {
+ public String getQueueNode(ServerName serverName, String queueId) {
return ZNodePaths.joinZNode(getRsNode(serverName), queueId);
}
diff --git
a/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationStorageBase.java
b/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationStorageBase.java
index 596167f..a239bf8 100644
---
a/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationStorageBase.java
+++
b/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationStorageBase.java
@@ -74,4 +74,8 @@ public abstract class ZKReplicationStorageBase {
throw new RuntimeException(e);
}
}
+
+ public ZKWatcher getZookeeper() {
+return this.zookeeper;
+ }
}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
index 9f229f4..02a897b 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
+++
b/hbase-server/src/main
[hbase] branch HBASE-24666 updated: HBASE-25071 ReplicationServer support start ReplicationSource internal (#2452)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch HBASE-24666
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/HBASE-24666 by this push:
new 3640775 HBASE-25071 ReplicationServer support start ReplicationSource
internal (#2452)
3640775 is described below
commit 36407759d67d67a089625c63cb3c3c81b624fe85
Author: Guanghao Zhang
AuthorDate: Mon Nov 9 11:46:02 2020 +0800
HBASE-25071 ReplicationServer support start ReplicationSource internal
(#2452)
Signed-off-by: XinSun
---
.../server/replication/ReplicationServer.proto | 14 +-
.../replication/ZKReplicationQueueStorage.java | 4 +-
.../replication/ZKReplicationStorageBase.java | 4 +
.../hadoop/hbase/master/MasterRpcServices.java | 2 +-
.../hadoop/hbase/regionserver/RSRpcServices.java | 6 +-
.../replication/HBaseReplicationEndpoint.java | 14 +-
.../hbase/replication/HReplicationServer.java | 175 ++---
.../replication/ReplicationServerRpcServices.java | 15 ++
.../regionserver/RecoveredReplicationSource.java | 9 +-
.../regionserver/ReplicationSource.java| 58 ++-
.../regionserver/ReplicationSourceFactory.java | 2 +-
.../regionserver/ReplicationSourceInterface.java | 6 +-
.../regionserver/ReplicationSourceManager.java | 4 +-
.../hbase/replication/ReplicationSourceDummy.java | 5 +-
.../hbase/replication/TestReplicationBase.java | 2 +-
.../replication/TestReplicationFetchServers.java | 43 +++--
...nServer.java => TestReplicationServerSink.java} | 24 +--
.../replication/TestReplicationServerSource.java | 69
.../regionserver/TestReplicationSource.java| 12 +-
.../regionserver/TestReplicationSourceManager.java | 16 +-
20 files changed, 395 insertions(+), 89 deletions(-)
diff --git
a/hbase-protocol-shaded/src/main/protobuf/server/replication/ReplicationServer.proto
b/hbase-protocol-shaded/src/main/protobuf/server/replication/ReplicationServer.proto
index ed334c4..925aed4 100644
---
a/hbase-protocol-shaded/src/main/protobuf/server/replication/ReplicationServer.proto
+++
b/hbase-protocol-shaded/src/main/protobuf/server/replication/ReplicationServer.proto
@@ -24,9 +24,21 @@ option java_generic_services = true;
option java_generate_equals_and_hash = true;
option optimize_for = SPEED;
+import "HBase.proto";
import "server/region/Admin.proto";
+message StartReplicationSourceRequest {
+ required ServerName server_name = 1;
+ required string queue_id = 2;
+}
+
+message StartReplicationSourceResponse {
+}
+
service ReplicationServerService {
rpc ReplicateWALEntry(ReplicateWALEntryRequest)
returns(ReplicateWALEntryResponse);
-}
\ No newline at end of file
+
+ rpc StartReplicationSource(StartReplicationSourceRequest)
+returns(StartReplicationSourceResponse);
+}
diff --git
a/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationQueueStorage.java
b/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationQueueStorage.java
index 6f1f5a3..2b9594e 100644
---
a/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationQueueStorage.java
+++
b/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationQueueStorage.java
@@ -80,7 +80,7 @@ import
org.apache.hbase.thirdparty.org.apache.commons.collections4.CollectionUti
*
*/
@InterfaceAudience.Private
-class ZKReplicationQueueStorage extends ZKReplicationStorageBase
+public class ZKReplicationQueueStorage extends ZKReplicationStorageBase
implements ReplicationQueueStorage {
private static final Logger LOG =
LoggerFactory.getLogger(ZKReplicationQueueStorage.class);
@@ -123,7 +123,7 @@ class ZKReplicationQueueStorage extends
ZKReplicationStorageBase
return ZNodePaths.joinZNode(queuesZNode, serverName.getServerName());
}
- private String getQueueNode(ServerName serverName, String queueId) {
+ public String getQueueNode(ServerName serverName, String queueId) {
return ZNodePaths.joinZNode(getRsNode(serverName), queueId);
}
diff --git
a/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationStorageBase.java
b/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationStorageBase.java
index 596167f..a239bf8 100644
---
a/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationStorageBase.java
+++
b/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationStorageBase.java
@@ -74,4 +74,8 @@ public abstract class ZKReplicationStorageBase {
throw new RuntimeException(e);
}
}
+
+ public ZKWatcher getZookeeper() {
+return this.zookeeper;
+ }
}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
b/hbase-server/src/main
[hbase] branch HBASE-25071-new updated (a28ec44 -> c8537ef)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch HBASE-25071-new in repository https://gitbox.apache.org/repos/asf/hbase.git. discard a28ec44 HBASE-25071 ReplicationServer support start ReplicationSource internal add c8537ef HBASE-25071 ReplicationServer support start ReplicationSource internal This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (a28ec44) \ N -- N -- N refs/heads/HBASE-25071-new (c8537ef) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../org/apache/hadoop/hbase/replication/HBaseReplicationEndpoint.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[hbase] branch HBASE-25071-new updated (7d4b041 -> a28ec44)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch HBASE-25071-new in repository https://gitbox.apache.org/repos/asf/hbase.git. discard 7d4b041 HBASE-25071 ReplicationServer support start ReplicationSource internal add a28ec44 HBASE-25071 ReplicationServer support start ReplicationSource internal This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (7d4b041) \ N -- N -- N refs/heads/HBASE-25071-new (a28ec44) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../replication/ZKReplicationQueueStorage.java | 4 +- .../replication/ZKReplicationStorageBase.java | 4 ++ .../hbase/replication/HReplicationServer.java | 2 + .../regionserver/ReplicationSource.java| 75 ++ 4 files changed, 56 insertions(+), 29 deletions(-)
[hbase] branch HBASE-25071-new updated (7d4b041 -> a28ec44)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch HBASE-25071-new in repository https://gitbox.apache.org/repos/asf/hbase.git. discard 7d4b041 HBASE-25071 ReplicationServer support start ReplicationSource internal add a28ec44 HBASE-25071 ReplicationServer support start ReplicationSource internal This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (7d4b041) \ N -- N -- N refs/heads/HBASE-25071-new (a28ec44) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../replication/ZKReplicationQueueStorage.java | 4 +- .../replication/ZKReplicationStorageBase.java | 4 ++ .../hbase/replication/HReplicationServer.java | 2 + .../regionserver/ReplicationSource.java| 75 ++ 4 files changed, 56 insertions(+), 29 deletions(-)
[hbase] branch HBASE-25071-new updated (7d4b041 -> a28ec44)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch HBASE-25071-new in repository https://gitbox.apache.org/repos/asf/hbase.git. discard 7d4b041 HBASE-25071 ReplicationServer support start ReplicationSource internal add a28ec44 HBASE-25071 ReplicationServer support start ReplicationSource internal This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (7d4b041) \ N -- N -- N refs/heads/HBASE-25071-new (a28ec44) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../replication/ZKReplicationQueueStorage.java | 4 +- .../replication/ZKReplicationStorageBase.java | 4 ++ .../hbase/replication/HReplicationServer.java | 2 + .../regionserver/ReplicationSource.java| 75 ++ 4 files changed, 56 insertions(+), 29 deletions(-)
[hbase] branch HBASE-25071-new updated (7d4b041 -> a28ec44)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch HBASE-25071-new in repository https://gitbox.apache.org/repos/asf/hbase.git. discard 7d4b041 HBASE-25071 ReplicationServer support start ReplicationSource internal add a28ec44 HBASE-25071 ReplicationServer support start ReplicationSource internal This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (7d4b041) \ N -- N -- N refs/heads/HBASE-25071-new (a28ec44) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../replication/ZKReplicationQueueStorage.java | 4 +- .../replication/ZKReplicationStorageBase.java | 4 ++ .../hbase/replication/HReplicationServer.java | 2 + .../regionserver/ReplicationSource.java| 75 ++ 4 files changed, 56 insertions(+), 29 deletions(-)
[hbase] branch HBASE-25071-new updated (97f0c23 -> 7d4b041)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch HBASE-25071-new in repository https://gitbox.apache.org/repos/asf/hbase.git. discard 97f0c23 HBASE-25071 ReplicationServer support start ReplicationSource internal add 7d4b041 HBASE-25071 ReplicationServer support start ReplicationSource internal This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (97f0c23) \ N -- N -- N refs/heads/HBASE-25071-new (7d4b041) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../org/apache/hadoop/hbase/replication/HBaseReplicationEndpoint.java | 2 +- .../java/org/apache/hadoop/hbase/replication/HReplicationServer.java| 2 +- .../apache/hadoop/hbase/replication/TestReplicationFetchServers.java| 2 -- 3 files changed, 2 insertions(+), 4 deletions(-)
[hbase] branch HBASE-25071-new updated (108dddd -> 97f0c23)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch HBASE-25071-new in repository https://gitbox.apache.org/repos/asf/hbase.git. discard 108 HBASE-25071 ReplicationServer support start ReplicationSource internal add 97f0c23 HBASE-25071 ReplicationServer support start ReplicationSource internal This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (108) \ N -- N -- N refs/heads/HBASE-25071-new (97f0c23) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../replication/HBaseReplicationEndpoint.java | 4 --- .../hbase/replication/HReplicationServer.java | 4 +-- .../replication/TestReplicationFetchServers.java | 41 +- .../replication/TestReplicationServerSink.java | 1 - 4 files changed, 35 insertions(+), 15 deletions(-)
[hbase] branch HBASE-25071-new updated (8a063c3 -> 108dddd)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch HBASE-25071-new in repository https://gitbox.apache.org/repos/asf/hbase.git. discard 8a063c3 HBASE-25071 ReplicationServer support start ReplicationSource internal add 108 HBASE-25071 ReplicationServer support start ReplicationSource internal This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (8a063c3) \ N -- N -- N refs/heads/HBASE-25071-new (108) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../hbase/replication/TestReplicationServerSink.java | 15 --- 1 file changed, 15 deletions(-)
[hbase] branch HBASE-25071-new updated (5a39f53 -> 8a063c3)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch HBASE-25071-new in repository https://gitbox.apache.org/repos/asf/hbase.git. discard 5a39f53 HBASE-25071 ReplicationServer support start ReplicationSource internal add 8a063c3 HBASE-25071 ReplicationServer support start ReplicationSource internal This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (5a39f53) \ N -- N -- N refs/heads/HBASE-25071-new (8a063c3) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../src/main/protobuf/server/master/Master.proto| 3 +-- .../apache/hadoop/hbase/master/MasterRpcServices.java | 6 ++ .../hbase/replication/HBaseReplicationEndpoint.java | 17 ++--- 3 files changed, 13 insertions(+), 13 deletions(-)
[hbase] branch HBASE-25071-new updated (bb46ef9 -> 5a39f53)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch HBASE-25071-new in repository https://gitbox.apache.org/repos/asf/hbase.git. discard bb46ef9 HBASE-25071 ReplicationServer support start ReplicationSource internal discard 6008ffe HBASE-24684 Fetch ReplicationSink servers list from HMaster instead o… (#2077) discard 8f8620d HBASE-24998 Introduce a ReplicationSourceOverallController interface and decouple ReplicationSourceManager and ReplicationSource (#2364) discard cd30e9b HBASE-24982 Disassemble the method replicateWALEntry from AdminService to a new interface ReplicationServerService (#2360) discard 4645d88 HBASE-24683 Add a basic ReplicationServer which only implement ReplicationSink Service (#2111) discard 0b5a777 HBASE-24735: Refactor ReplicationSourceManager: move logPositionAndCleanOldLogs/cleanUpHFileRefs to ReplicationSource inside (#2064) discard 430a813 HBASE-24681 Remove the cache walsById/walsByIdRecoveredQueues from ReplicationSourceManager (#2019) discard 9c8d54b HBASE-24682 Refactor ReplicationSource#addHFileRefs method: move it to ReplicationSourceManager (#2020) add 687e53b HBASE-24813 ReplicationSource should clear buffer usage on Replicatio… (#2191) add 4f8aa75 HBASE-25097 Wrong RIT page number in Master UI (#2458) add 4e59014 Add Zheng Wang to developers list. (#2457) add c86b065 HBASE-25085 Add support for java properties to hbase-vote.sh add 1093e34 HBASE-25088 CatalogFamilyFormat/MetaTableAccessor.parseRegionInfoFromRegionName incorrectly setEndKey to regionId (#2448) add 380585e HBASE-25098 ReplicationStatisticsChore runs in wrong time unit (#2460) add c312760 HBASE-25045 : Add 2.3.2 to the downloads page add b57bef5 HBASE-25077: hbck.jsp page loading fails, logs NPE in master log. (#2433) add 86557ed HBASE-25096 WAL size in RegionServer UI is wrong (#2456) add 80ffac2 HBASE-25070 : With new generic API getLogEntries, cleaning up unused RPC APIs add fbef545 HBASE-25017 Migrate flaky reporting jenkins job from Hadoop to hbase (#2466) add 68b56be HBASE-25100 conf and conn are assigned twice in HBaseReplicationEndpoint and HBaseInterClusterReplicationEndpoint (#2463) add 3aa612f Revert "HBASE-25017 Migrate flaky reporting jenkins job from Hadoop to hbase (#2466)" add 9d6af96 HBASE-25107 Migrate flaky reporting jenkins job from Hadoop to hbase (#2466) add 5d92662 HBASE-24967 The table.jsp cost long time to load if the table include… (#2326) add b268b1f HBASE-25103 Remove ZNodePaths.metaReplicaZNodes (#2464) add 9fc29c4 HBASE-24877 addendum: additional checks to avoid one extra possible race control in the initialize loop (#2400) add 0187607 HBASE-25109 Add MR Counters to WALPlayer; currently hard to tell if it is doing anything (#2468) add ee02e67 HBASE-25062 The link of "Re:(HBASE-451) Remove HTableDescriptor from HRegionInfo" invalid (#2455) add 1ce1cf6 HBASE-25132 Migrate flaky test jenkins job from Hadoop to hbase (#2485) add e663f09 HBASE-25133 Migrate HBase Nightly jenkins job from Hadoop to hbase (#2487) add 5351aca HBASE-25134 Migrate HBase PreCommit jenkins job from Hadoop to hbase (#2488) add e9de7d9 HBASE-24981 Enable table replication fails from 1.x to 2.x if table already exist at peer add a0b3d62 HBASE-25135 Convert the internal seperator while emitting the memstore read metrics to # (#2486) add 11a336a Revert "HBASE-25134 Migrate HBase PreCommit jenkins job from Hadoop to hbase (#2488)" add 3b91a15 HBASE-25091 Move LogComparator from ReplicationSource to AbstractFSWALProvider#.WALsStartTimeComparator (#2449) add 5e5166d HBASE-25120 Remove the deprecated annotation for MetaTableAccessor.getScanForTableName (#2493) add b0170d0 HBASE-25080 Should not use AssignmentManager to test whether a table is enabled or not (#2436) add 3226c17 HBASE-25115 HFilePrettyPrinter can't seek to the row which is the first row of a hfile add a8096b3 HBASE-25048 [HBCK2] Bypassed parent procedures are not updated in store (#2410) add 23ce918 HBASE-25147 : Serialize regionNames in ReopenTableRegionsProcedure add 9ba90e1 HBASE-25121 Refactor MetaTableAccessor.addRegionsToMeta and its usage places (#2476) add 16251db HBASE-25143 Remove branch-1.3 from precommit and docs (#2491) add 81f2cc5 HBASE-25154 Set java.io.tmpdir to project build directory to avoid writing std*deferred files to /tmp (#2502) add 9f238bd HBASE-24054 To be safe, jetty's version number should be blocked. add 14b523e HBASE-25146 Add extra logging at info level to HFileCorruptionChecker in order to report progress (#2503) add e885098 Revert "HBASE-24813 ReplicationSource should clear buffer usage on Replicatio… (#2191)" add a8c49a6 HBASE-25144 A
[hbase] branch HBASE-25071-new updated (bb46ef9 -> 5a39f53)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch HBASE-25071-new in repository https://gitbox.apache.org/repos/asf/hbase.git. discard bb46ef9 HBASE-25071 ReplicationServer support start ReplicationSource internal discard 6008ffe HBASE-24684 Fetch ReplicationSink servers list from HMaster instead o… (#2077) discard 8f8620d HBASE-24998 Introduce a ReplicationSourceOverallController interface and decouple ReplicationSourceManager and ReplicationSource (#2364) discard cd30e9b HBASE-24982 Disassemble the method replicateWALEntry from AdminService to a new interface ReplicationServerService (#2360) discard 4645d88 HBASE-24683 Add a basic ReplicationServer which only implement ReplicationSink Service (#2111) discard 0b5a777 HBASE-24735: Refactor ReplicationSourceManager: move logPositionAndCleanOldLogs/cleanUpHFileRefs to ReplicationSource inside (#2064) discard 430a813 HBASE-24681 Remove the cache walsById/walsByIdRecoveredQueues from ReplicationSourceManager (#2019) discard 9c8d54b HBASE-24682 Refactor ReplicationSource#addHFileRefs method: move it to ReplicationSourceManager (#2020) add 687e53b HBASE-24813 ReplicationSource should clear buffer usage on Replicatio… (#2191) add 4f8aa75 HBASE-25097 Wrong RIT page number in Master UI (#2458) add 4e59014 Add Zheng Wang to developers list. (#2457) add c86b065 HBASE-25085 Add support for java properties to hbase-vote.sh add 1093e34 HBASE-25088 CatalogFamilyFormat/MetaTableAccessor.parseRegionInfoFromRegionName incorrectly setEndKey to regionId (#2448) add 380585e HBASE-25098 ReplicationStatisticsChore runs in wrong time unit (#2460) add c312760 HBASE-25045 : Add 2.3.2 to the downloads page add b57bef5 HBASE-25077: hbck.jsp page loading fails, logs NPE in master log. (#2433) add 86557ed HBASE-25096 WAL size in RegionServer UI is wrong (#2456) add 80ffac2 HBASE-25070 : With new generic API getLogEntries, cleaning up unused RPC APIs add fbef545 HBASE-25017 Migrate flaky reporting jenkins job from Hadoop to hbase (#2466) add 68b56be HBASE-25100 conf and conn are assigned twice in HBaseReplicationEndpoint and HBaseInterClusterReplicationEndpoint (#2463) add 3aa612f Revert "HBASE-25017 Migrate flaky reporting jenkins job from Hadoop to hbase (#2466)" add 9d6af96 HBASE-25107 Migrate flaky reporting jenkins job from Hadoop to hbase (#2466) add 5d92662 HBASE-24967 The table.jsp cost long time to load if the table include… (#2326) add b268b1f HBASE-25103 Remove ZNodePaths.metaReplicaZNodes (#2464) add 9fc29c4 HBASE-24877 addendum: additional checks to avoid one extra possible race control in the initialize loop (#2400) add 0187607 HBASE-25109 Add MR Counters to WALPlayer; currently hard to tell if it is doing anything (#2468) add ee02e67 HBASE-25062 The link of "Re:(HBASE-451) Remove HTableDescriptor from HRegionInfo" invalid (#2455) add 1ce1cf6 HBASE-25132 Migrate flaky test jenkins job from Hadoop to hbase (#2485) add e663f09 HBASE-25133 Migrate HBase Nightly jenkins job from Hadoop to hbase (#2487) add 5351aca HBASE-25134 Migrate HBase PreCommit jenkins job from Hadoop to hbase (#2488) add e9de7d9 HBASE-24981 Enable table replication fails from 1.x to 2.x if table already exist at peer add a0b3d62 HBASE-25135 Convert the internal seperator while emitting the memstore read metrics to # (#2486) add 11a336a Revert "HBASE-25134 Migrate HBase PreCommit jenkins job from Hadoop to hbase (#2488)" add 3b91a15 HBASE-25091 Move LogComparator from ReplicationSource to AbstractFSWALProvider#.WALsStartTimeComparator (#2449) add 5e5166d HBASE-25120 Remove the deprecated annotation for MetaTableAccessor.getScanForTableName (#2493) add b0170d0 HBASE-25080 Should not use AssignmentManager to test whether a table is enabled or not (#2436) add 3226c17 HBASE-25115 HFilePrettyPrinter can't seek to the row which is the first row of a hfile add a8096b3 HBASE-25048 [HBCK2] Bypassed parent procedures are not updated in store (#2410) add 23ce918 HBASE-25147 : Serialize regionNames in ReopenTableRegionsProcedure add 9ba90e1 HBASE-25121 Refactor MetaTableAccessor.addRegionsToMeta and its usage places (#2476) add 16251db HBASE-25143 Remove branch-1.3 from precommit and docs (#2491) add 81f2cc5 HBASE-25154 Set java.io.tmpdir to project build directory to avoid writing std*deferred files to /tmp (#2502) add 9f238bd HBASE-24054 To be safe, jetty's version number should be blocked. add 14b523e HBASE-25146 Add extra logging at info level to HFileCorruptionChecker in order to report progress (#2503) add e885098 Revert "HBASE-24813 ReplicationSource should clear buffer usage on Replicatio… (#2191)" add a8c49a6 HBASE-25144 A
[hbase] branch HBASE-24666 updated: HBASE-24999 Master manages ReplicationServers (#2579)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch HBASE-24666
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/HBASE-24666 by this push:
new f67c3df HBASE-24999 Master manages ReplicationServers (#2579)
f67c3df is described below
commit f67c3dfc5abc558c944fa7e5319d03df3a05f4f3
Author: XinSun
AuthorDate: Wed Oct 28 18:59:57 2020 +0800
HBASE-24999 Master manages ReplicationServers (#2579)
Signed-off-by: Guanghao Zhang
---
.../server/master/ReplicationServerStatus.proto| 34
.../org/apache/hadoop/hbase/master/HMaster.java| 10 +
.../hadoop/hbase/master/MasterRpcServices.java | 37 +++-
.../apache/hadoop/hbase/master/MasterServices.java | 5 +
.../hbase/master/ReplicationServerManager.java | 204
.../replication/HBaseReplicationEndpoint.java | 150 +++
.../hbase/replication/HReplicationServer.java | 214 -
.../HBaseInterClusterReplicationEndpoint.java | 1 -
.../regionserver/ReplicationSyncUp.java| 4 +-
.../hbase/master/MockNoopMasterServices.java | 5 +
.../hbase/replication/TestReplicationBase.java | 2 +
.../hbase/replication/TestReplicationServer.java | 57 +-
12 files changed, 620 insertions(+), 103 deletions(-)
diff --git
a/hbase-protocol-shaded/src/main/protobuf/server/master/ReplicationServerStatus.proto
b/hbase-protocol-shaded/src/main/protobuf/server/master/ReplicationServerStatus.proto
new file mode 100644
index 000..d39a043
--- /dev/null
+++
b/hbase-protocol-shaded/src/main/protobuf/server/master/ReplicationServerStatus.proto
@@ -0,0 +1,34 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+syntax = "proto2";
+
+package hbase.pb;
+
+option java_package = "org.apache.hadoop.hbase.shaded.protobuf.generated";
+option java_outer_classname = "ReplicationServerStatusProtos";
+option java_generic_services = true;
+option java_generate_equals_and_hash = true;
+option optimize_for = SPEED;
+
+import "server/master/RegionServerStatus.proto";
+
+service ReplicationServerStatusService {
+
+ rpc ReplicationServerReport(RegionServerReportRequest)
+ returns(RegionServerReportResponse);
+}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
index b76b16c..2be9f98 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
@@ -359,6 +359,8 @@ public class HMaster extends HRegionServer implements
MasterServices {
// manager of assignment nodes in zookeeper
private AssignmentManager assignmentManager;
+ // server manager to deal with replication server info
+ private ReplicationServerManager replicationServerManager;
/**
* Cache for the meta region replica's locations. Also tracks their changes
to avoid stale
@@ -963,6 +965,8 @@ public class HMaster extends HRegionServer implements
MasterServices {
.collect(Collectors.toList());
this.assignmentManager.setupRIT(ritList);
+this.replicationServerManager = new ReplicationServerManager(this);
+
// Start RegionServerTracker with listing of servers found with exiting
SCPs -- these should
// be registered in the deadServers set -- and with the list of
servernames out on the
// filesystem that COULD BE 'alive' (we'll schedule SCPs for each and let
SCP figure it out).
@@ -1131,6 +1135,7 @@ public class HMaster extends HRegionServer implements
MasterServices {
this.hbckChore = new HbckChore(this);
getChoreService().scheduleChore(hbckChore);
this.serverManager.startChore();
+this.replicationServerManager.startChore();
// Only for rolling upgrade, where we need to migrate the data in
namespace table to meta table.
if (!waitForNamespaceOnline()) {
@@ -1390,6 +1395,11 @@ public class HMaster extends HRegionServer implements
MasterServices {
}
@Overrid
[hbase] branch branch-2 updated: HBASE-25128 RSGroupInfo's toString() and hashCode() does not take into account configuration map. (#2484)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-2 by this push:
new dd0a145 HBASE-25128 RSGroupInfo's toString() and hashCode() does not
take into account configuration map. (#2484)
dd0a145 is described below
commit dd0a14538bc30d750997117e75f4fe7e3ca6eff4
Author: sanjeetnishad95
AuthorDate: Fri Oct 23 06:12:00 2020 +0530
HBASE-25128 RSGroupInfo's toString() and hashCode() does not take into
account configuration map. (#2484)
Signed-off-by: Guanghao Zhang
---
.../src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java| 4
1 file changed, 4 insertions(+)
diff --git
a/hbase-common/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java
b/hbase-common/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java
index c96a80c..e183a61 100644
---
a/hbase-common/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java
+++
b/hbase-common/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java
@@ -164,6 +164,9 @@ public class RSGroupInfo {
sb.append(", ");
sb.append(" Tables:");
sb.append(this.tables);
+sb.append(", ");
+sb.append(" Configurations:");
+sb.append(this.configuration);
return sb.toString();
}
@@ -200,6 +203,7 @@ public class RSGroupInfo {
int result = servers.hashCode();
result = 31 * result + tables.hashCode();
result = 31 * result + name.hashCode();
+result = 31 * result + configuration.hashCode();
return result;
}
}
[hbase] branch master updated: HBASE-25128 RSGroupInfo's toString() and hashCode() does not take into account configuration map. (#2484)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/master by this push:
new d8de24c HBASE-25128 RSGroupInfo's toString() and hashCode() does not
take into account configuration map. (#2484)
d8de24c is described below
commit d8de24c31185cf325ad7fa96df4c455f4c817192
Author: sanjeetnishad95
AuthorDate: Fri Oct 23 06:12:00 2020 +0530
HBASE-25128 RSGroupInfo's toString() and hashCode() does not take into
account configuration map. (#2484)
Signed-off-by: Guanghao Zhang
---
.../src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java| 4
1 file changed, 4 insertions(+)
diff --git
a/hbase-common/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java
b/hbase-common/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java
index 1aa7ca1..bb4a4d7 100644
---
a/hbase-common/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java
+++
b/hbase-common/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfo.java
@@ -203,6 +203,9 @@ public class RSGroupInfo {
sb.append(", ");
sb.append(" Tables:");
sb.append(this.tables);
+sb.append(", ");
+sb.append(" Configurations:");
+sb.append(this.configuration);
return sb.toString();
}
@@ -239,6 +242,7 @@ public class RSGroupInfo {
int result = servers.hashCode();
result = 31 * result + tables.hashCode();
result = 31 * result + name.hashCode();
+result = 31 * result + configuration.hashCode();
return result;
}
}
[hbase] 03/07: HBASE-24735: Refactor ReplicationSourceManager: move logPositionAndCleanOldLogs/cleanUpHFileRefs to ReplicationSource inside (#2064)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch HBASE-24666
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 43410f9d21da4e711551604f8d5e3a9721e8d823
Author: Guanghao Zhang
AuthorDate: Tue Aug 11 20:07:09 2020 +0800
HBASE-24735: Refactor ReplicationSourceManager: move
logPositionAndCleanOldLogs/cleanUpHFileRefs to ReplicationSource inside (#2064)
Signed-off-by: Wellington Chevreuil
---
.../regionserver/RecoveredReplicationSource.java | 18 ++-
.../regionserver/ReplicationSource.java| 160 +++--
.../regionserver/ReplicationSourceInterface.java | 33 +++--
.../regionserver/ReplicationSourceManager.java | 158 ++--
.../regionserver/ReplicationSourceShipper.java | 17 +--
.../regionserver/ReplicationSourceWALReader.java | 17 +--
.../SerialReplicationSourceWALReader.java | 1 +
.../replication/regionserver/WALEntryBatch.java| 2 +-
.../hbase/replication/ReplicationSourceDummy.java | 24 ++--
.../regionserver/TestReplicationSource.java| 12 +-
.../regionserver/TestReplicationSourceManager.java | 50 +++
.../regionserver/TestWALEntryStream.java | 20 +--
12 files changed, 258 insertions(+), 254 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RecoveredReplicationSource.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RecoveredReplicationSource.java
index 46cf851..e3400ad 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RecoveredReplicationSource.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RecoveredReplicationSource.java
@@ -21,6 +21,7 @@ import java.io.IOException;
import java.util.List;
import java.util.UUID;
import java.util.concurrent.PriorityBlockingQueue;
+
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
@@ -44,15 +45,18 @@ public class RecoveredReplicationSource extends
ReplicationSource {
private static final Logger LOG =
LoggerFactory.getLogger(RecoveredReplicationSource.class);
+ private Path walDir;
+
private String actualPeerId;
@Override
- public void init(Configuration conf, FileSystem fs, ReplicationSourceManager
manager,
- ReplicationQueueStorage queueStorage, ReplicationPeer replicationPeer,
Server server,
- String peerClusterZnode, UUID clusterId, WALFileLengthProvider
walFileLengthProvider,
- MetricsSource metrics) throws IOException {
-super.init(conf, fs, manager, queueStorage, replicationPeer, server,
peerClusterZnode,
+ public void init(Configuration conf, FileSystem fs, Path walDir,
ReplicationSourceManager manager,
+ReplicationQueueStorage queueStorage, ReplicationPeer replicationPeer,
Server server,
+String peerClusterZnode, UUID clusterId, WALFileLengthProvider
walFileLengthProvider,
+MetricsSource metrics) throws IOException {
+super.init(conf, fs, walDir, manager, queueStorage, replicationPeer,
server, peerClusterZnode,
clusterId, walFileLengthProvider, metrics);
+this.walDir = walDir;
this.actualPeerId = this.replicationQueueInfo.getPeerId();
}
@@ -93,7 +97,7 @@ public class RecoveredReplicationSource extends
ReplicationSource {
deadRsDirectory.suffix(AbstractFSWALProvider.SPLITTING_EXT),
path.getName()) };
for (Path possibleLogLocation : locs) {
LOG.info("Possible location " +
possibleLogLocation.toUri().toString());
-if (manager.getFs().exists(possibleLogLocation)) {
+if (this.fs.exists(possibleLogLocation)) {
// We found the right new location
LOG.info("Log " + path + " still exists at " +
possibleLogLocation);
newPaths.add(possibleLogLocation);
@@ -126,7 +130,7 @@ public class RecoveredReplicationSource extends
ReplicationSource {
// N.B. the ReplicationSyncUp tool sets the manager.getWALDir to the root of
the wal
// area rather than to the wal area for a particular region server.
private Path getReplSyncUpPath(Path path) throws IOException {
-FileStatus[] rss = fs.listStatus(manager.getLogDir());
+FileStatus[] rss = fs.listStatus(walDir);
for (FileStatus rs : rss) {
Path p = rs.getPath();
FileStatus[] logs = fs.listStatus(p);
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
index a49cd5d..702a8a2 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource
[hbase] 05/07: HBASE-24982 Disassemble the method replicateWALEntry from AdminService to a new interface ReplicationServerService (#2360)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch HBASE-24666
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 8ed76d459704ffb01be681895cb6e3644a80b730
Author: XinSun
AuthorDate: Wed Sep 9 15:00:37 2020 +0800
HBASE-24982 Disassemble the method replicateWALEntry from AdminService to a
new interface ReplicationServerService (#2360)
Signed-off-by: Wellington Chevreuil
---
.../hadoop/hbase/client/AsyncConnectionImpl.java | 16 ++
.../server/replication/ReplicationServer.proto | 32
.../hadoop/hbase/replication/ReplicationUtils.java | 19 ++
.../hbase/client/AsyncClusterConnection.java | 5 +
.../hbase/client/AsyncClusterConnectionImpl.java | 5 +
.../hbase/client/AsyncReplicationServerAdmin.java | 80 +
.../hbase/protobuf/ReplicationProtobufUtil.java| 18 ++
.../hadoop/hbase/regionserver/RSRpcServices.java | 10 +-
.../replication/HBaseReplicationEndpoint.java | 57 +-
.../replication/ReplicationServerRpcServices.java | 200 +
.../HBaseInterClusterReplicationEndpoint.java | 7 +-
.../regionserver/ReplicationSource.java| 2 +-
.../hbase/client/DummyAsyncClusterConnection.java | 5 +
.../replication/TestHBaseReplicationEndpoint.java | 17 +-
.../hbase/replication/TestReplicationServer.java | 43 -
.../regionserver/TestReplicationSource.java| 2 +-
16 files changed, 291 insertions(+), 227 deletions(-)
diff --git
a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncConnectionImpl.java
b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncConnectionImpl.java
index 406af0d..e8f0fb0 100644
---
a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncConnectionImpl.java
+++
b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncConnectionImpl.java
@@ -66,6 +66,7 @@ import
org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos.AdminServic
import
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos.ClientService;
import org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos;
import
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.MasterService;
+import
org.apache.hadoop.hbase.shaded.protobuf.generated.ReplicationServerProtos.ReplicationServerService;
/**
* The implementation of AsyncConnection.
@@ -107,6 +108,8 @@ class AsyncConnectionImpl implements AsyncConnection {
private final ConcurrentMap rsStubs = new
ConcurrentHashMap<>();
private final ConcurrentMap adminSubs = new
ConcurrentHashMap<>();
+ private final ConcurrentMap
replStubs =
+ new ConcurrentHashMap<>();
private final AtomicReference masterStub = new
AtomicReference<>();
@@ -266,12 +269,25 @@ class AsyncConnectionImpl implements AsyncConnection {
return AdminService.newStub(rpcClient.createRpcChannel(serverName, user,
rpcTimeout));
}
+ private ReplicationServerService.Interface
createReplicationServerStub(ServerName serverName)
+ throws IOException {
+return ReplicationServerService.newStub(
+rpcClient.createRpcChannel(serverName, user, rpcTimeout));
+ }
+
AdminService.Interface getAdminStub(ServerName serverName) throws
IOException {
return ConcurrentMapUtils.computeIfAbsentEx(adminSubs,
getStubKey(AdminService.getDescriptor().getName(), serverName,
hostnameCanChange),
() -> createAdminServerStub(serverName));
}
+ ReplicationServerService.Interface getReplicationServerStub(ServerName
serverName)
+ throws IOException {
+return ConcurrentMapUtils.computeIfAbsentEx(replStubs,
+getStubKey(ReplicationServerService.Interface.class.getSimpleName(),
serverName,
+hostnameCanChange), () -> createReplicationServerStub(serverName));
+ }
+
CompletableFuture getMasterStub() {
return ConnectionUtils.getOrFetch(masterStub, masterStubMakeFuture, false,
() -> {
CompletableFuture future = new
CompletableFuture<>();
diff --git
a/hbase-protocol-shaded/src/main/protobuf/server/replication/ReplicationServer.proto
b/hbase-protocol-shaded/src/main/protobuf/server/replication/ReplicationServer.proto
new file mode 100644
index 000..ed334c4
--- /dev/null
+++
b/hbase-protocol-shaded/src/main/protobuf/server/replication/ReplicationServer.proto
@@ -0,0 +1,32 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed
[hbase] 02/07: HBASE-24681 Remove the cache walsById/walsByIdRecoveredQueues from ReplicationSourceManager (#2019)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch HBASE-24666
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 25764a3e44613c3c819693b35c2ee0398af0306d
Author: Guanghao Zhang
AuthorDate: Mon Jul 13 17:35:32 2020 +0800
HBASE-24681 Remove the cache walsById/walsByIdRecoveredQueues from
ReplicationSourceManager (#2019)
Signed-off-by: Wellington Chevreuil
---
.../regionserver/ReplicationSourceManager.java | 204 +++--
1 file changed, 62 insertions(+), 142 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
index 3869857..a222f4b 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
@@ -91,30 +91,6 @@ import
org.apache.hbase.thirdparty.com.google.common.util.concurrent.ThreadFacto
* No need synchronized on {@link #sources}. {@link #sources} is a
ConcurrentHashMap and there
* is a Lock for peer id in {@link PeerProcedureHandlerImpl}. So there is no
race for peer
* operations.
- * Need synchronized on {@link #walsById}. There are four methods which
modify it,
- * {@link #addPeer(String)}, {@link #removePeer(String)},
- * {@link #cleanOldLogs(String, boolean, ReplicationSourceInterface)} and
{@link #preLogRoll(Path)}.
- * {@link #walsById} is a ConcurrentHashMap and there is a Lock for peer id in
- * {@link PeerProcedureHandlerImpl}. So there is no race between {@link
#addPeer(String)} and
- * {@link #removePeer(String)}. {@link #cleanOldLogs(String, boolean,
ReplicationSourceInterface)}
- * is called by {@link ReplicationSourceInterface}. So no race with {@link
#addPeer(String)}.
- * {@link #removePeer(String)} will terminate the {@link
ReplicationSourceInterface} firstly, then
- * remove the wals from {@link #walsById}. So no race with {@link
#removePeer(String)}. The only
- * case need synchronized is {@link #cleanOldLogs(String, boolean,
ReplicationSourceInterface)} and
- * {@link #preLogRoll(Path)}.
- * No need synchronized on {@link #walsByIdRecoveredQueues}. There are
three methods which
- * modify it, {@link #removePeer(String)} ,
- * {@link #cleanOldLogs(String, boolean, ReplicationSourceInterface)} and
- * {@link ReplicationSourceManager.NodeFailoverWorker#run()}.
- * {@link #cleanOldLogs(String, boolean, ReplicationSourceInterface)} is
called by
- * {@link ReplicationSourceInterface}. {@link #removePeer(String)} will
terminate the
- * {@link ReplicationSourceInterface} firstly, then remove the wals from
- * {@link #walsByIdRecoveredQueues}. And {@link
ReplicationSourceManager.NodeFailoverWorker#run()}
- * will add the wals to {@link #walsByIdRecoveredQueues} firstly, then start
up a
- * {@link ReplicationSourceInterface}. So there is no race here. For
- * {@link ReplicationSourceManager.NodeFailoverWorker#run()} and {@link
#removePeer(String)}, there
- * is already synchronized on {@link #oldsources}. So no need synchronized on
- * {@link #walsByIdRecoveredQueues}.
* Need synchronized on {@link #latestPaths} to avoid the new open source
miss new log.
* Need synchronized on {@link #oldsources} to avoid adding recovered
source for the
* to-be-removed peer.
@@ -135,15 +111,6 @@ public class ReplicationSourceManager implements
ReplicationListener {
// All about stopping
private final Server server;
- // All logs we are currently tracking
- // Index structure of the map is: queue_id->logPrefix/logGroup->logs
- // For normal replication source, the peer id is same with the queue id
- private final ConcurrentMap>>
walsById;
- // Logs for recovered sources we are currently tracking
- // the map is: queue_id->logPrefix/logGroup->logs
- // For recovered source, the queue id's format is peer_id-servername-*
- private final ConcurrentMap>>
walsByIdRecoveredQueues;
-
private final SyncReplicationPeerMappingManager
syncReplicationPeerMappingManager;
private final Configuration conf;
@@ -199,8 +166,6 @@ public class ReplicationSourceManager implements
ReplicationListener {
this.replicationPeers = replicationPeers;
this.replicationTracker = replicationTracker;
this.server = server;
-this.walsById = new ConcurrentHashMap<>();
-this.walsByIdRecoveredQueues = new ConcurrentHashMap<>();
this.oldsources = new ArrayList<>();
this.conf = conf;
this.fs = fs;
@@ -338,7 +303,6 @@ public class ReplicationSourceManager implements
ReplicationListener {
// Delete queue from storage and memory and queue id is same with peer
id for normal
// source
deleteQueue(peerId);
- this.walsById.re
[hbase] 04/07: HBASE-24683 Add a basic ReplicationServer which only implement ReplicationSink Service (#2111)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch HBASE-24666
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit d43d2c9afa55139cb419229ba960941f823f4168
Author: XinSun
AuthorDate: Fri Sep 4 18:53:46 2020 +0800
HBASE-24683 Add a basic ReplicationServer which only implement
ReplicationSink Service (#2111)
Signed-off-by: Guanghao Zhang
---
.../java/org/apache/hadoop/hbase/util/DNS.java | 3 +-
.../hbase/replication/HReplicationServer.java | 391
.../replication/ReplicationServerRpcServices.java | 516 +
.../hbase/replication/TestReplicationServer.java | 151 ++
4 files changed, 1060 insertions(+), 1 deletion(-)
diff --git a/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DNS.java
b/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DNS.java
index 2b4e1cb..ddff6db 100644
--- a/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DNS.java
+++ b/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DNS.java
@@ -54,7 +54,8 @@ public final class DNS {
public enum ServerType {
MASTER("master"),
-REGIONSERVER("regionserver");
+REGIONSERVER("regionserver"),
+REPLICATIONSERVER("replicationserver");
private String name;
ServerType(String name) {
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HReplicationServer.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HReplicationServer.java
new file mode 100644
index 000..31dec0c
--- /dev/null
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HReplicationServer.java
@@ -0,0 +1,391 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.replication;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.ChoreService;
+import org.apache.hadoop.hbase.CoordinatedStateManager;
+import org.apache.hadoop.hbase.DoNotRetryIOException;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.Server;
+import org.apache.hadoop.hbase.ServerName;
+import org.apache.hadoop.hbase.client.AsyncClusterConnection;
+import org.apache.hadoop.hbase.client.ClusterConnectionFactory;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.log.HBaseMarkers;
+import org.apache.hadoop.hbase.regionserver.ReplicationService;
+import org.apache.hadoop.hbase.regionserver.ReplicationSinkService;
+import org.apache.hadoop.hbase.security.User;
+import org.apache.hadoop.hbase.security.UserProvider;
+import org.apache.hadoop.hbase.trace.TraceUtil;
+import org.apache.hadoop.hbase.util.Sleeper;
+import org.apache.hadoop.hbase.zookeeper.ZKWatcher;
+import org.apache.hadoop.util.ReflectionUtils;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * HReplicationServer which is responsible to all replication stuff. It checks
in with
+ * the HMaster. There are many HReplicationServers in a single HBase
deployment.
+ */
+@InterfaceAudience.Private
+@SuppressWarnings({ "deprecation"})
+public class HReplicationServer extends Thread implements Server {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(HReplicationServer.class);
+
+ /** replication server process name */
+ public static final String REPLICATION_SERVER = "replicationserver";
+
+ /**
+ * This servers start code.
+ */
+ protected final long startCode;
+
+ private volatile boolean stopped = false;
+
+ // Go down hard. Used if file system becomes unavailable and also in
+ // debugging and unit tests.
+ private AtomicBoolean abortRequested;
+
+ // flag set after we're done setting up server threads
+ final AtomicBoolean online = new AtomicBoolean(false);
+
+ /**
+ * The server name the Master sees us as. Its made from the hostname the
+ * master passes us, port, and server start code. Gets set after registration
+ * against Mas
[hbase] 07/07: HBASE-24684 Fetch ReplicationSink servers list from HMaster instead o… (#2077)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch HBASE-24666
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 1a64f1a0db1fb5184ca57cb062a3b0f1ca0e9865
Author: XinSun
AuthorDate: Sun Sep 20 10:54:43 2020 +0800
HBASE-24684 Fetch ReplicationSink servers list from HMaster instead o…
(#2077)
Signed-off-by: Wellington Chevreuil
---
.../src/main/protobuf/server/master/Master.proto | 12 +-
.../hadoop/hbase/coprocessor/MasterObserver.java | 16 +++
.../org/apache/hadoop/hbase/master/HMaster.java| 5 +
.../hadoop/hbase/master/MasterCoprocessorHost.java | 18 +++
.../hadoop/hbase/master/MasterRpcServices.java | 21 +++
.../apache/hadoop/hbase/master/MasterServices.java | 6 +
.../replication/HBaseReplicationEndpoint.java | 146 +++--
.../regionserver/ReplicationSource.java| 4 +-
.../hbase/master/MockNoopMasterServices.java | 5 +
.../replication/TestHBaseReplicationEndpoint.java | 5 +
.../replication/TestReplicationFetchServers.java | 106 +++
.../TestGlobalReplicationThrottler.java| 4 +
...stRegionReplicaReplicationEndpointNoMaster.java | 2 +
13 files changed, 334 insertions(+), 16 deletions(-)
diff --git a/hbase-protocol-shaded/src/main/protobuf/server/master/Master.proto
b/hbase-protocol-shaded/src/main/protobuf/server/master/Master.proto
index 118ce77..7dec566 100644
--- a/hbase-protocol-shaded/src/main/protobuf/server/master/Master.proto
+++ b/hbase-protocol-shaded/src/main/protobuf/server/master/Master.proto
@@ -717,6 +717,13 @@ message BalancerDecisionsResponse {
repeated BalancerDecision balancer_decision = 1;
}
+message ListReplicationSinkServersRequest {
+}
+
+message ListReplicationSinkServersResponse {
+ repeated ServerName server_name = 1;
+}
+
service MasterService {
/** Used by the client to get the number of regions that have received the
updated schema */
rpc GetSchemaAlterStatus(GetSchemaAlterStatusRequest)
@@ -1146,10 +1153,13 @@ service MasterService {
returns (RenameRSGroupResponse);
rpc UpdateRSGroupConfig(UpdateRSGroupConfigRequest)
- returns (UpdateRSGroupConfigResponse);
+returns (UpdateRSGroupConfigResponse);
rpc GetLogEntries(LogRequest)
returns(LogEntry);
+
+ rpc ListReplicationSinkServers(ListReplicationSinkServersRequest)
+returns (ListReplicationSinkServersResponse);
}
// HBCK Service definitions.
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
index ac35caa..ec009cc 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
@@ -1782,4 +1782,20 @@ public interface MasterObserver {
default void
postHasUserPermissions(ObserverContext ctx,
String userName, List permissions) throws IOException {
}
+
+ /**
+ * Called before getting servers for replication sink.
+ * @param ctx the coprocessor instance's environment
+ */
+ default void
preListReplicationSinkServers(ObserverContext ctx)
+throws IOException {
+ }
+
+ /**
+ * Called after getting servers for replication sink.
+ * @param ctx the coprocessor instance's environment
+ */
+ default void
postListReplicationSinkServers(ObserverContext
ctx)
+throws IOException {
+ }
}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
index f58096f..b76b16c 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
@@ -3856,4 +3856,9 @@ public class HMaster extends HRegionServer implements
MasterServices {
}
return compactionState;
}
+
+ @Override
+ public List listReplicationSinkServers() throws IOException {
+return this.serverManager.getOnlineServersList();
+ }
}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
index 01d1a62..f775eba 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
@@ -2038,4 +2038,22 @@ public class MasterCoprocessorHost
}
});
}
+
+ public void preListReplicationSinkServers() throws IOException {
+execOperation(coprocEnvironments.isEmpty() ? null : new
MasterObserverOperation() {
+ @Override
+ public void call(MasterObserver observer) throws IOException {
+observer.preListReplicationSinkServers(this);
+ }
+});
+ }
+
+ p
[hbase] 01/07: HBASE-24682 Refactor ReplicationSource#addHFileRefs method: move it to ReplicationSourceManager (#2020)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch HBASE-24666
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 7a41a3ee3cc6a2070ed0f598524d02662cebf548
Author: Guanghao Zhang
AuthorDate: Wed Jul 8 14:29:08 2020 +0800
HBASE-24682 Refactor ReplicationSource#addHFileRefs method: move it to
ReplicationSourceManager (#2020)
Signed-off-by: Wellington Chevreuil
---
.../regionserver/ReplicationSource.java| 38 +
.../regionserver/ReplicationSourceInterface.java | 14 ---
.../regionserver/ReplicationSourceManager.java | 48 +-
.../hbase/replication/ReplicationSourceDummy.java | 9 +---
4 files changed, 49 insertions(+), 60 deletions(-)
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
index 8091d0c..a49cd5d 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
@@ -27,7 +27,6 @@ import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
-import java.util.Set;
import java.util.TreeMap;
import java.util.UUID;
import java.util.concurrent.ConcurrentHashMap;
@@ -37,6 +36,7 @@ import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicLong;
import java.util.function.Predicate;
+
import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
@@ -46,21 +46,17 @@ import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.Server;
import org.apache.hadoop.hbase.ServerName;
import org.apache.hadoop.hbase.TableDescriptors;
-import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.regionserver.HRegionServer;
import org.apache.hadoop.hbase.regionserver.RSRpcServices;
import org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost;
import org.apache.hadoop.hbase.replication.ChainWALEntryFilter;
import org.apache.hadoop.hbase.replication.ClusterMarkingEntryFilter;
import org.apache.hadoop.hbase.replication.ReplicationEndpoint;
-import org.apache.hadoop.hbase.replication.ReplicationException;
import org.apache.hadoop.hbase.replication.ReplicationPeer;
import org.apache.hadoop.hbase.replication.ReplicationQueueInfo;
import org.apache.hadoop.hbase.replication.ReplicationQueueStorage;
import org.apache.hadoop.hbase.replication.SystemTableWALEntryFilter;
import org.apache.hadoop.hbase.replication.WALEntryFilter;
-import org.apache.hadoop.hbase.util.Bytes;
-import org.apache.hadoop.hbase.util.Pair;
import org.apache.hadoop.hbase.util.Threads;
import org.apache.hadoop.hbase.wal.AbstractFSWALProvider;
import org.apache.hadoop.hbase.wal.WAL.Entry;
@@ -281,38 +277,6 @@ public class ReplicationSource implements
ReplicationSourceInterface {
}
}
- @Override
- public void addHFileRefs(TableName tableName, byte[] family, List> pairs)
- throws ReplicationException {
-String peerId = replicationPeer.getId();
-Set namespaces = replicationPeer.getNamespaces();
-Map> tableCFMap = replicationPeer.getTableCFs();
-if (tableCFMap != null) { // All peers with TableCFs
- List tableCfs = tableCFMap.get(tableName);
- if (tableCFMap.containsKey(tableName)
- && (tableCfs == null || tableCfs.contains(Bytes.toString(family {
-this.queueStorage.addHFileRefs(peerId, pairs);
-metrics.incrSizeOfHFileRefsQueue(pairs.size());
- } else {
-LOG.debug("HFiles will not be replicated belonging to the table {}
family {} to peer id {}",
-tableName, Bytes.toString(family), peerId);
- }
-} else if (namespaces != null) { // Only for set NAMESPACES peers
- if (namespaces.contains(tableName.getNamespaceAsString())) {
-this.queueStorage.addHFileRefs(peerId, pairs);
-metrics.incrSizeOfHFileRefsQueue(pairs.size());
- } else {
-LOG.debug("HFiles will not be replicated belonging to the table {}
family {} to peer id {}",
-tableName, Bytes.toString(family), peerId);
- }
-} else {
- // user has explicitly not defined any table cfs for replication, means
replicate all the
- // data
- this.queueStorage.addHFileRefs(peerId, pairs);
- metrics.incrSizeOfHFileRefsQueue(pairs.size());
-}
- }
-
private ReplicationEndpoint createReplicationEndpoint()
throws InstantiationException, IllegalAccessException,
ClassNotFoundException, IOException {
RegionServerCoprocessorHost rsServerHost = null;
diff --git
a/hbase-server/src/main/java/o
[hbase] branch HBASE-24666 updated (827ac96 -> 1a64f1a)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch HBASE-24666 in repository https://gitbox.apache.org/repos/asf/hbase.git. omit 827ac96 HBASE-24684 Fetch ReplicationSink servers list from HMaster instead o… (#2077) omit 17621c8 HBASE-24998 Introduce a ReplicationSourceOverallController interface and decouple ReplicationSourceManager and ReplicationSource (#2364) omit 56cd97f HBASE-24982 Disassemble the method replicateWALEntry from AdminService to a new interface ReplicationServerService (#2360) omit 6625e92 HBASE-24683 Add a basic ReplicationServer which only implement ReplicationSink Service (#2111) omit 5a62429 HBASE-24735: Refactor ReplicationSourceManager: move logPositionAndCleanOldLogs/cleanUpHFileRefs to ReplicationSource inside (#2064) omit 5646555 HBASE-24681 Remove the cache walsById/walsByIdRecoveredQueues from ReplicationSourceManager (#2019) omit c315e68 HBASE-24682 Refactor ReplicationSource#addHFileRefs method: move it to ReplicationSourceManager (#2020) add 3aa612f Revert "HBASE-25017 Migrate flaky reporting jenkins job from Hadoop to hbase (#2466)" add 9d6af96 HBASE-25107 Migrate flaky reporting jenkins job from Hadoop to hbase (#2466) add 5d92662 HBASE-24967 The table.jsp cost long time to load if the table include… (#2326) add b268b1f HBASE-25103 Remove ZNodePaths.metaReplicaZNodes (#2464) add 9fc29c4 HBASE-24877 addendum: additional checks to avoid one extra possible race control in the initialize loop (#2400) add 0187607 HBASE-25109 Add MR Counters to WALPlayer; currently hard to tell if it is doing anything (#2468) add ee02e67 HBASE-25062 The link of "Re:(HBASE-451) Remove HTableDescriptor from HRegionInfo" invalid (#2455) add 1ce1cf6 HBASE-25132 Migrate flaky test jenkins job from Hadoop to hbase (#2485) add e663f09 HBASE-25133 Migrate HBase Nightly jenkins job from Hadoop to hbase (#2487) add 5351aca HBASE-25134 Migrate HBase PreCommit jenkins job from Hadoop to hbase (#2488) add e9de7d9 HBASE-24981 Enable table replication fails from 1.x to 2.x if table already exist at peer add a0b3d62 HBASE-25135 Convert the internal seperator while emitting the memstore read metrics to # (#2486) add 11a336a Revert "HBASE-25134 Migrate HBase PreCommit jenkins job from Hadoop to hbase (#2488)" add 3b91a15 HBASE-25091 Move LogComparator from ReplicationSource to AbstractFSWALProvider#.WALsStartTimeComparator (#2449) add 5e5166d HBASE-25120 Remove the deprecated annotation for MetaTableAccessor.getScanForTableName (#2493) add b0170d0 HBASE-25080 Should not use AssignmentManager to test whether a table is enabled or not (#2436) add 3226c17 HBASE-25115 HFilePrettyPrinter can't seek to the row which is the first row of a hfile add a8096b3 HBASE-25048 [HBCK2] Bypassed parent procedures are not updated in store (#2410) add 23ce918 HBASE-25147 : Serialize regionNames in ReopenTableRegionsProcedure add 9ba90e1 HBASE-25121 Refactor MetaTableAccessor.addRegionsToMeta and its usage places (#2476) add 16251db HBASE-25143 Remove branch-1.3 from precommit and docs (#2491) add 81f2cc5 HBASE-25154 Set java.io.tmpdir to project build directory to avoid writing std*deferred files to /tmp (#2502) add 9f238bd HBASE-24054 To be safe, jetty's version number should be blocked. add 14b523e HBASE-25146 Add extra logging at info level to HFileCorruptionChecker in order to report progress (#2503) add e885098 Revert "HBASE-24813 ReplicationSource should clear buffer usage on Replicatio… (#2191)" add a8c49a6 HBASE-25144 Add Hadoop-3.3.0 to personality hadoopcheck (#2492) add 3537f91 HBASE-23742 Document that with split-to-hfile data over the MOB threshold will be treated as normal data (#2505) add 2fc79e2 HBASE-23959 Fix javadoc for JDK11 (#2500) add aff8bbf HBASE-25160 Refactor AccessController and VisibilityController (#2506) add 0d63318 HBASE-25124 Support changing region replica count without disabling table (#2497) add 8a995ae HBASE-25165 Change 'State time' in UI so sorts (#2508) add bdcafa8 HBASE-24025: Improve performance of move_servers_rsgroup by using async region move API (#1549) add b82d8a5 HBASE-25156 TestMasterFailover.testSimpleMasterFailover is flaky (#2507) add 78ae1f1 HBASE-24628 Region normalizer now respects a rate limit add 665a876 HBASE-22976 [HBCK2] Add RecoveredEditsPlayer (#2504) add f34ba06 HBASE-25163 Increase the timeout value for nightly jobs (#2512) add b815207 HBASE-25171 Remove ZNodePaths.namespaceZNode (#2526) add c367e91 HBASE-25162 Make flaky tests run more aggressively (#2525) add accd975 HBASE-25065 WAL archival to be done by a separate thread (#2501) add 5a47444 H
[hbase] 06/07: HBASE-24998 Introduce a ReplicationSourceOverallController interface and decouple ReplicationSourceManager and ReplicationSource (#2364)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch HBASE-24666
in repository https://gitbox.apache.org/repos/asf/hbase.git
commit 4f77158eaa8cf99c5d1b8b56ca1736287bbc6f5f
Author: Guanghao Zhang
AuthorDate: Sun Sep 20 09:02:53 2020 +0800
HBASE-24998 Introduce a ReplicationSourceOverallController interface and
decouple ReplicationSourceManager and ReplicationSource (#2364)
Signed-off-by: meiyi
---
.../java/org/apache/hadoop/hbase/HConstants.java | 2 +
.../hbase/replication/ReplicationListener.java | 2 +-
.../replication/ReplicationSourceController.java | 31 +++--
.../regionserver/RecoveredReplicationSource.java | 18 ++-
.../regionserver/ReplicationSource.java| 31 ++---
.../regionserver/ReplicationSourceInterface.java | 25 ++--
.../regionserver/ReplicationSourceManager.java | 141 +++--
.../regionserver/ReplicationSourceWALReader.java | 13 +-
.../hbase/replication/ReplicationSourceDummy.java | 21 ++-
.../regionserver/TestReplicationSourceManager.java | 11 +-
.../regionserver/TestWALEntryStream.java | 15 ++-
11 files changed, 167 insertions(+), 143 deletions(-)
diff --git a/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
b/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
index e1d3de9..5ffd1d6 100644
--- a/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
+++ b/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
@@ -979,6 +979,8 @@ public final class HConstants {
/*
* cluster replication constants.
*/
+ public static final String REPLICATION_OFFLOAD_ENABLE_KEY =
"hbase.replication.offload.enabled";
+ public static final boolean REPLICATION_OFFLOAD_ENABLE_DEFAULT = false;
public static final String
REPLICATION_SOURCE_SERVICE_CLASSNAME =
"hbase.replication.source.service";
public static final String REPLICATION_SERVICE_CLASSNAME_DEFAULT =
diff --git
a/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ReplicationListener.java
b/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ReplicationListener.java
index f040bf9..6ecbb46 100644
---
a/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ReplicationListener.java
+++
b/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ReplicationListener.java
@@ -33,5 +33,5 @@ public interface ReplicationListener {
* A region server has been removed from the local cluster
* @param regionServer the removed region server
*/
- public void regionServerRemoved(String regionServer);
+ void regionServerRemoved(String regionServer);
}
diff --git
a/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ReplicationListener.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ReplicationSourceController.java
similarity index 50%
copy from
hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ReplicationListener.java
copy to
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ReplicationSourceController.java
index f040bf9..5bb9dd6 100644
---
a/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ReplicationListener.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ReplicationSourceController.java
@@ -1,5 +1,4 @@
-/*
- *
+/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
@@ -18,20 +17,32 @@
*/
package org.apache.hadoop.hbase.replication;
+import java.util.concurrent.atomic.AtomicLong;
+
+import
org.apache.hadoop.hbase.replication.regionserver.MetricsReplicationGlobalSourceSource;
+import
org.apache.hadoop.hbase.replication.regionserver.RecoveredReplicationSource;
import org.apache.yetus.audience.InterfaceAudience;
/**
- * The replication listener interface can be implemented if a class needs to
subscribe to events
- * generated by the ReplicationTracker. These events include things like
addition/deletion of peer
- * clusters or failure of a local region server. To receive events, the class
also needs to register
- * itself with a Replication Tracker.
+ * Used to control all replication sources inside one RegionServer or
ReplicationServer.
+ * Used by {@link
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource} or
+ * {@link RecoveredReplicationSource}.
*/
@InterfaceAudience.Private
-public interface ReplicationListener {
+public interface ReplicationSourceController {
+
+ /**
+ * Returns the maximum size in bytes of edits held in memory which are
pending replication
+ * across all sources inside this RegionServer or ReplicationServer.
+ */
+ long getTotalBufferLimit();
+
+ AtomicLong getTotalBufferUsed();
+
+ MetricsReplicationGlobalSourceSource getGlobalMetrics
[hbase] branch branch-1.4 updated: HBASE-25204 Nightly job failed as the name of jdk and maven changed (#2567)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch branch-1.4
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-1.4 by this push:
new cb5c88c HBASE-25204 Nightly job failed as the name of jdk and maven
changed (#2567)
cb5c88c is described below
commit cb5c88c779df796ad894f31b3d26d4fbdde27a77
Author: Guanghao Zhang
AuthorDate: Tue Oct 20 16:58:43 2020 +0800
HBASE-25204 Nightly job failed as the name of jdk and maven changed (#2567)
Signed-off-by: Duo Zhang
---
dev-support/Jenkinsfile | 4 ++--
dev-support/adhoc_run_tests/Jenkinsfile | 4 ++--
dev-support/flaky-tests/run-flaky-tests.Jenkinsfile | 4 ++--
3 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/dev-support/Jenkinsfile b/dev-support/Jenkinsfile
index 66405d0..d1240b6 100644
--- a/dev-support/Jenkinsfile
+++ b/dev-support/Jenkinsfile
@@ -443,9 +443,9 @@ curl -L -o personality.sh "${env.PROJECT_PERSONALITY}"
// See http://hbase.apache.org/book.html#maven.release
stage ('create source tarball') {
tools {
-maven 'Maven (latest)'
+maven 'maven_latest'
// this needs to be set to the jdk that ought to be used to build
releases on the branch the Jenkinsfile is stored in.
-jdk "JDK 1.7 (latest)"
+jdk "jdk_1.7_latest"
}
environment {
BASEDIR = "${env.WORKSPACE}/component"
diff --git a/dev-support/adhoc_run_tests/Jenkinsfile
b/dev-support/adhoc_run_tests/Jenkinsfile
index e06fdba..476795d 100644
--- a/dev-support/adhoc_run_tests/Jenkinsfile
+++ b/dev-support/adhoc_run_tests/Jenkinsfile
@@ -51,10 +51,10 @@ pipeline {
stages {
stage ('run tests') {
tools {
-maven 'Maven (latest)'
+maven 'maven_latest'
// this needs to be set to the jdk that ought to be used to build
releases on the branch
// the Jenkinsfile is stored in.
-jdk "JDK 1.8 (latest)"
+jdk "jdk_1.8_latest"
}
steps {
sh """#!/bin/bash -e
diff --git a/dev-support/flaky-tests/run-flaky-tests.Jenkinsfile
b/dev-support/flaky-tests/run-flaky-tests.Jenkinsfile
index c84a3ac..a2fb06a 100644
--- a/dev-support/flaky-tests/run-flaky-tests.Jenkinsfile
+++ b/dev-support/flaky-tests/run-flaky-tests.Jenkinsfile
@@ -34,8 +34,8 @@ pipeline {
}
tools {
// this should match what the yetus nightly job for the branch will use
-maven 'Maven (latest)'
-jdk "JDK 1.8 (latest)"
+maven 'maven_latest'
+jdk "jdk_1.8_latest"
}
stages {
stage ('run flaky tests') {
[hbase] branch branch-1 updated: HBASE-25204 Nightly job failed as the name of jdk and maven changed (#2567)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch branch-1
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-1 by this push:
new 8e387e9 HBASE-25204 Nightly job failed as the name of jdk and maven
changed (#2567)
8e387e9 is described below
commit 8e387e9dfa4c97b8ae3da1e0d5d1ee7a6cf4fdb3
Author: Guanghao Zhang
AuthorDate: Tue Oct 20 16:58:43 2020 +0800
HBASE-25204 Nightly job failed as the name of jdk and maven changed (#2567)
Signed-off-by: Duo Zhang
---
dev-support/Jenkinsfile | 4 ++--
dev-support/adhoc_run_tests/Jenkinsfile | 4 ++--
dev-support/flaky-tests/run-flaky-tests.Jenkinsfile | 4 ++--
3 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/dev-support/Jenkinsfile b/dev-support/Jenkinsfile
index 66405d0..d1240b6 100644
--- a/dev-support/Jenkinsfile
+++ b/dev-support/Jenkinsfile
@@ -443,9 +443,9 @@ curl -L -o personality.sh "${env.PROJECT_PERSONALITY}"
// See http://hbase.apache.org/book.html#maven.release
stage ('create source tarball') {
tools {
-maven 'Maven (latest)'
+maven 'maven_latest'
// this needs to be set to the jdk that ought to be used to build
releases on the branch the Jenkinsfile is stored in.
-jdk "JDK 1.7 (latest)"
+jdk "jdk_1.7_latest"
}
environment {
BASEDIR = "${env.WORKSPACE}/component"
diff --git a/dev-support/adhoc_run_tests/Jenkinsfile
b/dev-support/adhoc_run_tests/Jenkinsfile
index e06fdba..476795d 100644
--- a/dev-support/adhoc_run_tests/Jenkinsfile
+++ b/dev-support/adhoc_run_tests/Jenkinsfile
@@ -51,10 +51,10 @@ pipeline {
stages {
stage ('run tests') {
tools {
-maven 'Maven (latest)'
+maven 'maven_latest'
// this needs to be set to the jdk that ought to be used to build
releases on the branch
// the Jenkinsfile is stored in.
-jdk "JDK 1.8 (latest)"
+jdk "jdk_1.8_latest"
}
steps {
sh """#!/bin/bash -e
diff --git a/dev-support/flaky-tests/run-flaky-tests.Jenkinsfile
b/dev-support/flaky-tests/run-flaky-tests.Jenkinsfile
index 6d7d121..c048d83 100644
--- a/dev-support/flaky-tests/run-flaky-tests.Jenkinsfile
+++ b/dev-support/flaky-tests/run-flaky-tests.Jenkinsfile
@@ -34,8 +34,8 @@ pipeline {
}
tools {
// this should match what the yetus nightly job for the branch will use
-maven 'Maven (latest)'
-jdk "JDK 1.8 (latest)"
+maven 'maven_latest'
+jdk "jdk_1.8_latest"
}
stages {
stage ('run flaky tests') {
[hbase] branch branch-2.2 updated: HBASE-25204 Nightly job failed as the name of jdk and maven changed (#2567)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch branch-2.2
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-2.2 by this push:
new 2e244a3 HBASE-25204 Nightly job failed as the name of jdk and maven
changed (#2567)
2e244a3 is described below
commit 2e244a3bdc7071e7b35df80b597a8c3cda65f774
Author: Guanghao Zhang
AuthorDate: Tue Oct 20 16:58:43 2020 +0800
HBASE-25204 Nightly job failed as the name of jdk and maven changed (#2567)
Signed-off-by: Duo Zhang
---
dev-support/Jenkinsfile | 4 ++--
dev-support/adhoc_run_tests/Jenkinsfile | 4 ++--
dev-support/flaky-tests/run-flaky-tests.Jenkinsfile | 4 ++--
3 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/dev-support/Jenkinsfile b/dev-support/Jenkinsfile
index e34ce29..4fc26ec 100644
--- a/dev-support/Jenkinsfile
+++ b/dev-support/Jenkinsfile
@@ -505,9 +505,9 @@ pipeline {
// See http://hbase.apache.org/book.html#maven.release
stage ('packaging and integration') {
tools {
-maven 'Maven (latest)'
+maven 'maven_latest'
// this needs to be set to the jdk that ought to be used to build
releases on the branch the Jenkinsfile is stored in.
-jdk "JDK 1.8 (latest)"
+jdk "jdk_1.8_latest"
}
environment {
BASEDIR = "${env.WORKSPACE}/component"
diff --git a/dev-support/adhoc_run_tests/Jenkinsfile
b/dev-support/adhoc_run_tests/Jenkinsfile
index e06fdba..476795d 100644
--- a/dev-support/adhoc_run_tests/Jenkinsfile
+++ b/dev-support/adhoc_run_tests/Jenkinsfile
@@ -51,10 +51,10 @@ pipeline {
stages {
stage ('run tests') {
tools {
-maven 'Maven (latest)'
+maven 'maven_latest'
// this needs to be set to the jdk that ought to be used to build
releases on the branch
// the Jenkinsfile is stored in.
-jdk "JDK 1.8 (latest)"
+jdk "jdk_1.8_latest"
}
steps {
sh """#!/bin/bash -e
diff --git a/dev-support/flaky-tests/run-flaky-tests.Jenkinsfile
b/dev-support/flaky-tests/run-flaky-tests.Jenkinsfile
index 6d7d121..c048d83 100644
--- a/dev-support/flaky-tests/run-flaky-tests.Jenkinsfile
+++ b/dev-support/flaky-tests/run-flaky-tests.Jenkinsfile
@@ -34,8 +34,8 @@ pipeline {
}
tools {
// this should match what the yetus nightly job for the branch will use
-maven 'Maven (latest)'
-jdk "JDK 1.8 (latest)"
+maven 'maven_latest'
+jdk "jdk_1.8_latest"
}
stages {
stage ('run flaky tests') {
[hbase] branch master updated (7d00e43 -> 28f2c17)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hbase.git. from 7d00e43 HBASE-25166 MobFileCompactionChore is closing the master's shared cluster connection (#2514) add 28f2c17 HBASE-25204 Nightly job failed as the name of jdk and maven changed (#2567) No new revisions were added by this update. Summary of changes: dev-support/Jenkinsfile | 4 ++-- dev-support/adhoc_run_tests/Jenkinsfile | 4 ++-- dev-support/flaky-tests/run-flaky-tests.Jenkinsfile | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-)
[hbase] branch branch-2.3 updated: HBASE-25204 Nightly job failed as the name of jdk and maven changed (#2567)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch branch-2.3
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-2.3 by this push:
new 7c4c3d0 HBASE-25204 Nightly job failed as the name of jdk and maven
changed (#2567)
7c4c3d0 is described below
commit 7c4c3d06853c0317a22bef6da7081dbb7c102f56
Author: Guanghao Zhang
AuthorDate: Tue Oct 20 16:58:43 2020 +0800
HBASE-25204 Nightly job failed as the name of jdk and maven changed (#2567)
Signed-off-by: Duo Zhang
---
dev-support/Jenkinsfile | 4 ++--
dev-support/adhoc_run_tests/Jenkinsfile | 4 ++--
dev-support/flaky-tests/run-flaky-tests.Jenkinsfile | 4 ++--
3 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/dev-support/Jenkinsfile b/dev-support/Jenkinsfile
index d2aaa9d..bb6c57c 100644
--- a/dev-support/Jenkinsfile
+++ b/dev-support/Jenkinsfile
@@ -590,9 +590,9 @@ pipeline {
// TODO (HBASE-23870): replace this with invocation of the release tool
stage ('packaging and integration') {
tools {
-maven 'Maven (latest)'
+maven 'maven_latest'
// this needs to be set to the jdk that ought to be used to build
releases on the branch the Jenkinsfile is stored in.
-jdk "JDK 1.8 (latest)"
+jdk "jdk_1.8_latest"
}
environment {
BASEDIR = "${env.WORKSPACE}/component"
diff --git a/dev-support/adhoc_run_tests/Jenkinsfile
b/dev-support/adhoc_run_tests/Jenkinsfile
index e06fdba..476795d 100644
--- a/dev-support/adhoc_run_tests/Jenkinsfile
+++ b/dev-support/adhoc_run_tests/Jenkinsfile
@@ -51,10 +51,10 @@ pipeline {
stages {
stage ('run tests') {
tools {
-maven 'Maven (latest)'
+maven 'maven_latest'
// this needs to be set to the jdk that ought to be used to build
releases on the branch
// the Jenkinsfile is stored in.
-jdk "JDK 1.8 (latest)"
+jdk "jdk_1.8_latest"
}
steps {
sh """#!/bin/bash -e
diff --git a/dev-support/flaky-tests/run-flaky-tests.Jenkinsfile
b/dev-support/flaky-tests/run-flaky-tests.Jenkinsfile
index b92347f..0ba200b 100644
--- a/dev-support/flaky-tests/run-flaky-tests.Jenkinsfile
+++ b/dev-support/flaky-tests/run-flaky-tests.Jenkinsfile
@@ -34,8 +34,8 @@ pipeline {
}
tools {
// this should match what the yetus nightly job for the branch will use
-maven 'Maven (latest)'
-jdk "JDK 1.8 (latest)"
+maven 'maven_latest'
+jdk "jdk_1.8_latest"
}
stages {
stage ('run flaky tests') {
[hbase] branch branch-2 updated: HBASE-25204 Nightly job failed as the name of jdk and maven changed (#2567)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/branch-2 by this push:
new aa888f7 HBASE-25204 Nightly job failed as the name of jdk and maven
changed (#2567)
aa888f7 is described below
commit aa888f7e3885c77d4289a54bcbfff85b137a3d07
Author: Guanghao Zhang
AuthorDate: Tue Oct 20 16:58:43 2020 +0800
HBASE-25204 Nightly job failed as the name of jdk and maven changed (#2567)
Signed-off-by: Duo Zhang
---
dev-support/Jenkinsfile | 4 ++--
dev-support/adhoc_run_tests/Jenkinsfile | 4 ++--
dev-support/flaky-tests/run-flaky-tests.Jenkinsfile | 4 ++--
3 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/dev-support/Jenkinsfile b/dev-support/Jenkinsfile
index d2aaa9d..bb6c57c 100644
--- a/dev-support/Jenkinsfile
+++ b/dev-support/Jenkinsfile
@@ -590,9 +590,9 @@ pipeline {
// TODO (HBASE-23870): replace this with invocation of the release tool
stage ('packaging and integration') {
tools {
-maven 'Maven (latest)'
+maven 'maven_latest'
// this needs to be set to the jdk that ought to be used to build
releases on the branch the Jenkinsfile is stored in.
-jdk "JDK 1.8 (latest)"
+jdk "jdk_1.8_latest"
}
environment {
BASEDIR = "${env.WORKSPACE}/component"
diff --git a/dev-support/adhoc_run_tests/Jenkinsfile
b/dev-support/adhoc_run_tests/Jenkinsfile
index e06fdba..476795d 100644
--- a/dev-support/adhoc_run_tests/Jenkinsfile
+++ b/dev-support/adhoc_run_tests/Jenkinsfile
@@ -51,10 +51,10 @@ pipeline {
stages {
stage ('run tests') {
tools {
-maven 'Maven (latest)'
+maven 'maven_latest'
// this needs to be set to the jdk that ought to be used to build
releases on the branch
// the Jenkinsfile is stored in.
-jdk "JDK 1.8 (latest)"
+jdk "jdk_1.8_latest"
}
steps {
sh """#!/bin/bash -e
diff --git a/dev-support/flaky-tests/run-flaky-tests.Jenkinsfile
b/dev-support/flaky-tests/run-flaky-tests.Jenkinsfile
index b92347f..0ba200b 100644
--- a/dev-support/flaky-tests/run-flaky-tests.Jenkinsfile
+++ b/dev-support/flaky-tests/run-flaky-tests.Jenkinsfile
@@ -34,8 +34,8 @@ pipeline {
}
tools {
// this should match what the yetus nightly job for the branch will use
-maven 'Maven (latest)'
-jdk "JDK 1.8 (latest)"
+maven 'maven_latest'
+jdk "jdk_1.8_latest"
}
stages {
stage ('run flaky tests') {
[hbase] branch master updated (7d00e43 -> 28f2c17)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hbase.git. from 7d00e43 HBASE-25166 MobFileCompactionChore is closing the master's shared cluster connection (#2514) add 28f2c17 HBASE-25204 Nightly job failed as the name of jdk and maven changed (#2567) No new revisions were added by this update. Summary of changes: dev-support/Jenkinsfile | 4 ++-- dev-support/adhoc_run_tests/Jenkinsfile | 4 ++-- dev-support/flaky-tests/run-flaky-tests.Jenkinsfile | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-)
[hbase] branch master updated (7d00e43 -> 28f2c17)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hbase.git. from 7d00e43 HBASE-25166 MobFileCompactionChore is closing the master's shared cluster connection (#2514) add 28f2c17 HBASE-25204 Nightly job failed as the name of jdk and maven changed (#2567) No new revisions were added by this update. Summary of changes: dev-support/Jenkinsfile | 4 ++-- dev-support/adhoc_run_tests/Jenkinsfile | 4 ++-- dev-support/flaky-tests/run-flaky-tests.Jenkinsfile | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-)
[hbase] branch master updated (7d00e43 -> 28f2c17)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hbase.git. from 7d00e43 HBASE-25166 MobFileCompactionChore is closing the master's shared cluster connection (#2514) add 28f2c17 HBASE-25204 Nightly job failed as the name of jdk and maven changed (#2567) No new revisions were added by this update. Summary of changes: dev-support/Jenkinsfile | 4 ++-- dev-support/adhoc_run_tests/Jenkinsfile | 4 ++-- dev-support/flaky-tests/run-flaky-tests.Jenkinsfile | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-)
[hbase] branch master updated (7d00e43 -> 28f2c17)
This is an automated email from the ASF dual-hosted git repository. zghao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hbase.git. from 7d00e43 HBASE-25166 MobFileCompactionChore is closing the master's shared cluster connection (#2514) add 28f2c17 HBASE-25204 Nightly job failed as the name of jdk and maven changed (#2567) No new revisions were added by this update. Summary of changes: dev-support/Jenkinsfile | 4 ++-- dev-support/adhoc_run_tests/Jenkinsfile | 4 ++-- dev-support/flaky-tests/run-flaky-tests.Jenkinsfile | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-)
[hbase] branch master updated: "HBASE-25144 Add Hadoop-3.3.0 to personality hadoopcheck" try again (#2536)
This is an automated email from the ASF dual-hosted git repository.
zghao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase.git
The following commit(s) were added to refs/heads/master by this push:
new bc20203 "HBASE-25144 Add Hadoop-3.3.0 to personality hadoopcheck" try
again (#2536)
bc20203 is described below
commit bc20203965c06e12f7061ec6c40221ccee220ae3
Author: Nick Dimiduk
AuthorDate: Mon Oct 12 19:18:47 2020 -0700
"HBASE-25144 Add Hadoop-3.3.0 to personality hadoopcheck" try again (#2536)
Due to HBASE-23834, HBASE-19256, HADOOP-16152, HBase 2.2.x and 2.3.x
cannot run on Hadoop 3.3.0, or any Hadoop version that has upgraded to
Jetty 9.4.
Signed-off-by: Sean Busbey
Signed-off-by: stack
Signed-off-by: Guanghao Zhang
Signed-off-by: Duo Zhang
---
dev-support/hbase-personality.sh | 9 -
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/dev-support/hbase-personality.sh b/dev-support/hbase-personality.sh
index 95bbd44..9b39ead 100755
--- a/dev-support/hbase-personality.sh
+++ b/dev-support/hbase-personality.sh
@@ -609,8 +609,15 @@ function hadoopcheck_rebuild
else
hbase_hadoop3_versions="3.0.3 3.1.1 3.1.2"
fi
+ elif [[ "${PATCH_BRANCH}" = branch-2.2 ]] || [[ "${PATCH_BRANCH}" =
branch-2.3 ]]; then
+yetus_info "Setting Hadoop 3 versions to test based on
branch-2.2/branch-2.3 rules"
+if [[ "${QUICK_HADOOPCHECK}" == "true" ]]; then
+ hbase_hadoop3_versions="3.1.2 3.2.1"
+else
+ hbase_hadoop3_versions="3.1.1 3.1.2 3.2.0 3.2.1"
+fi
else
-yetus_info "Setting Hadoop 3 versions to test based on
branch-2.2+/master/feature branch rules"
+yetus_info "Setting Hadoop 3 versions to test based on
branch-2.4+/master/feature branch rules"
if [[ "${QUICK_HADOOPCHECK}" == "true" ]]; then
hbase_hadoop3_versions="3.1.2 3.2.1 3.3.0"
else
