[
https://issues.apache.org/jira/browse/HBASE-22839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917084#comment-16917084
]
Andrew Purtell edited comment on HBASE-22839 at 8/27/19 8:28 PM:
-----------------------------------------------------------------
It doesn't matter if the development is done in branch-1 or master first, you
can do whatever is more convenient. Eventually we will need patches for master,
branch-2, and branch-1 before they can all be committed, though.
However we don't need a complete solution of different selectable replication
models normalized over three branches right now. What we need, as claimed by
this issue, is an option that ensures batches within one region are shipped to
the sink clusters in order, to be committed to master, branch-2, and branch-1.
If something isn't implemented soon, there is no way it will make 1.5.
was (Author: apurtell):
It doesn't matter if the development is done in branch-1 or master first, you
can do whatever is more convenient. Eventually we will need patches for master,
branch-2, and branch-1 before they can all be committed, though.
However we don't need a complete solution of different selectable replication
models normalized over three branches right now. What we need, as claimed by
this issue, is an option that ensures batches within one region are shipped to
the sink clusters in orde, to be committed to master, branch-2, and branch-1.
If something isn't implemented soon, there is no way it will make 1.5.
> Make sure the batches within one region are shipped to the sink clusters in
> order (branch-1)
> --------------------------------------------------------------------------------------------
>
> Key: HBASE-22839
> URL: https://issues.apache.org/jira/browse/HBASE-22839
> Project: HBase
> Issue Type: Improvement
> Components: Replication
> Affects Versions: 1.3.4, 1.3.5
> Reporter: Bin Shi
> Assignee: Bin Shi
> Priority: Major
> Fix For: 1.5.0
>
>
> Problem Statement:
> In the cross-cluster replication validation, we found some cells in source
> and sink cluster can have the same row key, the same timestamp but different
> values. The happens when mutations with the same row key are submitted in
> batch without specifying the timestamp, and the same timestamp in the unit of
> millisecond is assigned at the time when they are committed to the WAL.
> When this happens, if the major compaction hasn’t happened yet and you scan
> the table, you can find some cells have the same row key, the same timestamps
> but different values, like the first three rows in the following table.
> |Row Key 1|CF0::Column 1|Timestatmp 1|Value 1|
> |Row Key 1|CF0::Column 1|Timestatmp 1|Value 2|
> |Row Key 1|CF0::Column 1|Timestatmp 1|Value 3|
> |Row Key 2|CF0::Column 1|Timestatmp 2|Value 4|
> |Row Key 3|CF0::Column 1|Timestatmp 4|Value 5|
> The ordering of the first three rows is indeterminate in the presence of the
> cross-replication, so after compaction, in the master cluster you will see
> “Row Key 1, CF0::Column1, Timestamp1” having the value 3, but in the slave
> cluster, you might see the cell having one of the three possible values 1, 2,
> 3, which results in data inconsistency issue between the master and slave
> clusters.
> Root Cause Analysis:
> In HBaseInterClusterReplicationEndpoint.createBatches() of branch-1.3, the
> WAL entries from the same region could be split into different batches
> according to replication RPC limit and these batches are shipped by
> ReplicationSource concurrently, so the batches for the same region could
> arrive at the sink in the slave clusters then apply to the region
> synchronously in indeterminate order.
> Solution:
> In HBase 3.0.0 and 2.1.0, [~Apache9]&[~openinx]&[~fenghh] provided Serial
> Replication HBASE-20046 which guarantees the order of pushing logs to slave
> clusters is same as the order of requests from client in the master cluster.
> It contains mainly two changes:
> # Recording the replication "barriers" in ZooKeeper to synchronize the
> replication across old/failed RS and new RS to provide strict ordering
> semantics even in the presence of region-move or RS failure.
> # Make sure the batches within one region are shipped to the slave clusters
> in order.
> The second part of change is exactly what we need and the minimal change to
> fix the issue in this JIRA.
> To fix the issue in this JIRA, we have two options:
> # Cherry-Pick HBASE-20046 to branch 1.3. Pros: It also fixes the data
> inconsistency issue when there is region-move or RS failure and help to
> reduce the noises in our cross-cluster replication/backup validation which is
> our ultimate goal. Cons: the change is big and I'm not sure for now whether
> the change is self-contained or it has other dependencies which need to port
> to branch 1.3 too; and we need longer time to validate and stabilize.
> # Port the minimal change or make the equivalent change as the second part
> of HBASE-20046 to make sure the batches within one region are shipped to the
> slave clusters in order."
> With limited knowledge about HBase Release Schedule and Process, I prefer
> option 2 because of cons of option 1, but I'm open to option 1 and other
> options. Thoughts?
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
