[jira] [Created] (HBASE-27463) Reset sizeOfLogQueue when refresh replication source
ruanhui created HBASE-27463: --- Summary: Reset sizeOfLogQueue when refresh replication source Key: HBASE-27463 URL: https://issues.apache.org/jira/browse/HBASE-27463 Project: HBase Issue Type: Bug Components: Replication Affects Versions: 3.0.0-alpha-3 Reporter: ruanhui Assignee: ruanhui Fix For: 3.0.0-alpha-4 When refresh replication sources, we don't clear the metric. That may cause the value of sizeOfLogQueue metric wrong. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27100) Add documentation for Replication Observability Framework in hbase book.
[ https://issues.apache.org/jira/browse/HBASE-27100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani resolved HBASE-27100. -- Fix Version/s: (was: 3.0.0-alpha-4) Resolution: Implemented > Add documentation for Replication Observability Framework in hbase book. > > > Key: HBASE-27100 > URL: https://issues.apache.org/jira/browse/HBASE-27100 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Rushabh Shah >Assignee: Rushabh Shah >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (HBASE-27100) Add documentation for Replication Observability Framework in hbase book.
[ https://issues.apache.org/jira/browse/HBASE-27100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani reopened HBASE-27100: -- > Add documentation for Replication Observability Framework in hbase book. > > > Key: HBASE-27100 > URL: https://issues.apache.org/jira/browse/HBASE-27100 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Rushabh Shah >Assignee: Rushabh Shah >Priority: Major > Fix For: 3.0.0-alpha-4 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27085) Create REPLICATION_SINK_TRACKER table to persist sentinel rows coming from source cluster.
[ https://issues.apache.org/jira/browse/HBASE-27085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani resolved HBASE-27085. -- Resolution: Implemented > Create REPLICATION_SINK_TRACKER table to persist sentinel rows coming from > source cluster. > -- > > Key: HBASE-27085 > URL: https://issues.apache.org/jira/browse/HBASE-27085 > Project: HBase > Issue Type: Sub-task >Reporter: Rushabh Shah >Assignee: Rushabh Shah >Priority: Major > Fix For: 3.0.0-alpha-4 > > > This work is to create sink tracker table to persist tracker rows coming from > replication source cluster. > Create ReplicationMarkerChore to create replication marker rows periodically. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-26925) Create WAL event tracker table to track all the WAL events.
[ https://issues.apache.org/jira/browse/HBASE-26925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani resolved HBASE-26925. -- Fix Version/s: (was: 3.0.0-alpha-4) Resolution: Implemented > Create WAL event tracker table to track all the WAL events. > --- > > Key: HBASE-26925 > URL: https://issues.apache.org/jira/browse/HBASE-26925 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: Rushabh Shah >Assignee: Rushabh Shah >Priority: Major > > Design Doc: > [https://docs.google.com/document/d/14oZ5ssY28hvJaQD_Jg9kWX7LfUKUyyU2PCA93PPzVko/edit#] > Create wal event tracker table to track WAL events. Whenever we roll the WAL, > we will save the WAL name, WAL length, region server, timestamp in a table. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (HBASE-27085) Create REPLICATION_SINK_TRACKER table to persist sentinel rows coming from source cluster.
[ https://issues.apache.org/jira/browse/HBASE-27085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani reopened HBASE-27085: -- > Create REPLICATION_SINK_TRACKER table to persist sentinel rows coming from > source cluster. > -- > > Key: HBASE-27085 > URL: https://issues.apache.org/jira/browse/HBASE-27085 > Project: HBase > Issue Type: Sub-task >Reporter: Rushabh Shah >Assignee: Rushabh Shah >Priority: Major > Fix For: 3.0.0-alpha-4 > > > This work is to create sink tracker table to persist tracker rows coming from > replication source cluster. > Create ReplicationMarkerChore to create replication marker rows periodically. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (HBASE-26925) Create WAL event tracker table to track all the WAL events.
[ https://issues.apache.org/jira/browse/HBASE-26925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani reopened HBASE-26925: -- > Create WAL event tracker table to track all the WAL events. > --- > > Key: HBASE-26925 > URL: https://issues.apache.org/jira/browse/HBASE-26925 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: Rushabh Shah >Assignee: Rushabh Shah >Priority: Major > Fix For: 3.0.0-alpha-4 > > > Design Doc: > [https://docs.google.com/document/d/14oZ5ssY28hvJaQD_Jg9kWX7LfUKUyyU2PCA93PPzVko/edit#] > Create wal event tracker table to track WAL events. Whenever we roll the WAL, > we will save the WAL name, WAL length, region server, timestamp in a table. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] Merge feature branch HBASE-26913 to master/branch-2
Congratulations to everyone, especially the performers who implemented the new framework! On Thu, Nov 3, 2022 at 11:07 AM Rushabh Shah wrote: > With 3 +1 and no -1, we will merge this PR and will create a backport PR > for branch-2. Thank you everyone. > > > Rushabh Shah > >- Software Engineering LMTS | Salesforce >- > - Mobile: 213 422 9052 > > > > On Wed, Nov 2, 2022 at 7:18 PM 张铎(Duo Zhang) > wrote: > > > After the recent update of the design and implementation, change my > > vote to +1. Detailed can be found in the discussion on the PR. > > > > https://github.com/apache/hbase/pull/4556 > > > > So we have gotten 3 +1s for merging back the feature branch to master. > > > > Thanks. > > > > 张铎(Duo Zhang) 于2022年6月22日周三 12:10写道： > > > > > > > > Left some comments on the PR. > > > > > > PTAL. > > > > > > Thanks. > > > > > > 张铎(Duo Zhang) 于2022年6月22日周三 10:08写道： > > >> > > >> Skimmed the design doc, I think it is a very useful feature for > > detecting and debugging replication inconsistencies. Good job. > > >> > > >> And since the feature is default off, I'm optimistic that we can merge > > it first and then keep polishing it in the future, as it will not affect > > the normal write and replication flow. > > >> > > >> Let me take a look at the PR and report back here. The first > suggestion > > is that, please use lower case table names, as all our system tables are > in > > lower case :) > > >> > > >> Thanks for the great work. > > >> > > >> Andrew Purtell 于2022年6月22日周三 06:29写道： > > >>> > > >>> +1 > > >>> > > >>> > On Jun 21, 2022, at 10:07 AM, Rushabh Shah < > > rushabh.s...@salesforce.com.invalid> wrote: > > >>> > > > >>> > Hi everyone, > > >>> > > > >>> > I'm writing this request to propose a merge of HBASE-26913[1] > > "Replication > > >>> > Observability Framework" to master and branch-2. > > >>> > The goal is to persist replication related metadata to newly > created > > hbase > > >>> > tables to reduce MTTD (Mean Time To Detect) and > > >>> > MTTR (Mean Time To Repair) replication inconsistencies between > > Primary and > > >>> > DR clusters. > > >>> > The design doc is here[2]. > > >>> > > > >>> > We have created 2 new hbase tables, REPLICATION.WALEVENTTRACKER and > > >>> > REPLICATION.SINK_TRACKER. The first table > > >>> > will store all the WAL events (ACTIVE, ROLLING, ROLLED) along with > > metadata > > >>> > (wal name, wal length, region server name, timestamp) > > >>> > from all region servers. We have introduced a new chore called > > >>> > ReplicationMarkerChore which will create special marker/sentinel > rows > > >>> > periodically (which is configurable) and inject them directly to > > WAL. This > > >>> > marker rows will be handled specially and replicated to > > >>> > sink cluster and will be persisted to REPLICATION.SINK_TRACKER > table. > > >>> > > > >>> > Highlights > > >>> > * The entire feature is configurable. Defaults to false. > > >>> > * A new section is added to the HBase book which covers the feature > > and how > > >>> > to use it. > > >>> > > > >>> > The vote will open for at least 72 hours. > > >>> > Please vote: > > >>> > [+1]Merge the changes from HBASE-26913 to master/branch-2 > > >>> > [+/-0] Neutral > > >>> > [-1]Disagree (please include actionable feedback) > > >>> > > > >>> > 1. https://issues.apache.org/jira/browse/HBASE-26913 > > >>> > 2. > > >>> > > > > https://docs.google.com/document/d/14oZ5ssY28hvJaQD_Jg9kWX7LfUKUyyU2PCA93PPzVko/edit#heading=h.9oum2kn0zj5r > > >>> > > > >>> > > > >>> > Thanks, > > >>> > Rushabh > > > -- Best regards, Andrew Unrest, ignorance distilled, nihilistic imbeciles - It's what we’ve earned Welcome, apocalypse, what’s taken you so long? Bring us the fitting end that we’ve been counting on - A23, Welcome, Apocalypse
[jira] [Resolved] (HBASE-27100) Add documentation for Replication Observability Framework in hbase book.
[ https://issues.apache.org/jira/browse/HBASE-27100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh Shah resolved HBASE-27100. -- Fix Version/s: 3.0.0-alpha-4 Resolution: Fixed > Add documentation for Replication Observability Framework in hbase book. > > > Key: HBASE-27100 > URL: https://issues.apache.org/jira/browse/HBASE-27100 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Rushabh Shah >Assignee: Rushabh Shah >Priority: Major > Fix For: 3.0.0-alpha-4 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-26925) Create WAL event tracker table to track all the WAL events.
[ https://issues.apache.org/jira/browse/HBASE-26925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh Shah resolved HBASE-26925. -- Fix Version/s: 3.0.0-alpha-4 Resolution: Fixed > Create WAL event tracker table to track all the WAL events. > --- > > Key: HBASE-26925 > URL: https://issues.apache.org/jira/browse/HBASE-26925 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: Rushabh Shah >Assignee: Rushabh Shah >Priority: Major > Fix For: 3.0.0-alpha-4 > > > Design Doc: > [https://docs.google.com/document/d/14oZ5ssY28hvJaQD_Jg9kWX7LfUKUyyU2PCA93PPzVko/edit#] > Create wal event tracker table to track WAL events. Whenever we roll the WAL, > we will save the WAL name, WAL length, region server, timestamp in a table. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27085) Create REPLICATION_SINK_TRACKER table to persist sentinel rows coming from source cluster.
[ https://issues.apache.org/jira/browse/HBASE-27085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh Shah resolved HBASE-27085. -- Fix Version/s: 3.0.0-alpha-4 Resolution: Fixed > Create REPLICATION_SINK_TRACKER table to persist sentinel rows coming from > source cluster. > -- > > Key: HBASE-27085 > URL: https://issues.apache.org/jira/browse/HBASE-27085 > Project: HBase > Issue Type: Sub-task >Reporter: Rushabh Shah >Assignee: Rushabh Shah >Priority: Major > Fix For: 3.0.0-alpha-4 > > > This work is to create sink tracker table to persist tracker rows coming from > replication source cluster. > Create ReplicationMarkerChore to create replication marker rows periodically. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] Merge feature branch HBASE-26913 to master/branch-2
With 3 +1 and no -1, we will merge this PR and will create a backport PR for branch-2. Thank you everyone. Rushabh Shah - Software Engineering LMTS | Salesforce - - Mobile: 213 422 9052 On Wed, Nov 2, 2022 at 7:18 PM 张铎(Duo Zhang) wrote: > After the recent update of the design and implementation, change my > vote to +1. Detailed can be found in the discussion on the PR. > > https://github.com/apache/hbase/pull/4556 > > So we have gotten 3 +1s for merging back the feature branch to master. > > Thanks. > > 张铎(Duo Zhang) 于2022年6月22日周三 12:10写道： > > > > > Left some comments on the PR. > > > > PTAL. > > > > Thanks. > > > > 张铎(Duo Zhang) 于2022年6月22日周三 10:08写道： > >> > >> Skimmed the design doc, I think it is a very useful feature for > detecting and debugging replication inconsistencies. Good job. > >> > >> And since the feature is default off, I'm optimistic that we can merge > it first and then keep polishing it in the future, as it will not affect > the normal write and replication flow. > >> > >> Let me take a look at the PR and report back here. The first suggestion > is that, please use lower case table names, as all our system tables are in > lower case :) > >> > >> Thanks for the great work. > >> > >> Andrew Purtell 于2022年6月22日周三 06:29写道： > >>> > >>> +1 > >>> > >>> > On Jun 21, 2022, at 10:07 AM, Rushabh Shah < > rushabh.s...@salesforce.com.invalid> wrote: > >>> > > >>> > Hi everyone, > >>> > > >>> > I'm writing this request to propose a merge of HBASE-26913[1] > "Replication > >>> > Observability Framework" to master and branch-2. > >>> > The goal is to persist replication related metadata to newly created > hbase > >>> > tables to reduce MTTD (Mean Time To Detect) and > >>> > MTTR (Mean Time To Repair) replication inconsistencies between > Primary and > >>> > DR clusters. > >>> > The design doc is here[2]. > >>> > > >>> > We have created 2 new hbase tables, REPLICATION.WALEVENTTRACKER and > >>> > REPLICATION.SINK_TRACKER. The first table > >>> > will store all the WAL events (ACTIVE, ROLLING, ROLLED) along with > metadata > >>> > (wal name, wal length, region server name, timestamp) > >>> > from all region servers. We have introduced a new chore called > >>> > ReplicationMarkerChore which will create special marker/sentinel rows > >>> > periodically (which is configurable) and inject them directly to > WAL. This > >>> > marker rows will be handled specially and replicated to > >>> > sink cluster and will be persisted to REPLICATION.SINK_TRACKER table. > >>> > > >>> > Highlights > >>> > * The entire feature is configurable. Defaults to false. > >>> > * A new section is added to the HBase book which covers the feature > and how > >>> > to use it. > >>> > > >>> > The vote will open for at least 72 hours. > >>> > Please vote: > >>> > [+1]Merge the changes from HBASE-26913 to master/branch-2 > >>> > [+/-0] Neutral > >>> > [-1]Disagree (please include actionable feedback) > >>> > > >>> > 1. https://issues.apache.org/jira/browse/HBASE-26913 > >>> > 2. > >>> > > https://docs.google.com/document/d/14oZ5ssY28hvJaQD_Jg9kWX7LfUKUyyU2PCA93PPzVko/edit#heading=h.9oum2kn0zj5r > >>> > > >>> > > >>> > Thanks, > >>> > Rushabh >
[jira] [Resolved] (HBASE-27460) Fix the hadolint errors after HBASE-27456
[ https://issues.apache.org/jira/browse/HBASE-27460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-27460. --- Fix Version/s: 2.6.0 3.0.0-alpha-4 2.5.2 2.4.16 Hadoop Flags: Reviewed Resolution: Fixed Pushed to branch-2.4+. Thanks [~zghao] for reviewing! > Fix the hadolint errors after HBASE-27456 > - > > Key: HBASE-27460 > URL: https://issues.apache.org/jira/browse/HBASE-27460 > Project: HBase > Issue Type: Task > Components: scripts >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.6.0, 3.0.0-alpha-4, 2.5.2, 2.4.16 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
