Hi Hbase community, On our production installation we have two hbase clusters in two different datacenters.The primary datacenter replicates the data to the secondary datacenter.When we create the tables, we first create on the secondary datacenter and then on the primary and then we set replication scope to 1 on the primary.The peer pointing to quorum zk of the secondary cluster is configured on the primary. Initially, replication worked fine and data was replicated.We have recently noticed that some tables are empty in the secondary datacenter. So most likely the data is no longer replicated. I'm seeing lines like this in the logs:
Recovered source for cluster/machine(s) replicav1: Total replicated edits: 0, current progress:walGroup [db11%2C16020%2C1637849866921]: currently replicating from: hdfs://rozzanohadoopcluster/hbase/oldWALs/db11-hd%2C16020%2C1637849866921.1637849874263 at position: -1 Recovered source for cluster/machine(s) replicav1: Total replicated edits: 0, current progress:walGroup [db09%2C16020%2C1637589840862]: currently replicating from: hdfs://rozzanohadoopcluster/hbase/oldWALs/db09-hd%2C16020%2C1637589840862.1637589846870 at position: -1 Recovered source for cluster/machine(s) replicav1: Total replicated edits: 0, current progress:walGroup [db13%2C16020%2C1635424806449]: currently replicating from: hdfs://rozzanohadoopcluster/hbase/oldWALs/db13%2C16020%2C1635424806449.1635424812985 at position: -1 2021-12-12 09:13:47,148 INFO [rzv-db09-hd:16020Replication Statistics #0] regionserver.Replication: ormal source for cluster replicav1: Total replicated edits: 0, current progress:walGroup [db09%2C16020%2C1638791923537]: currently replicating from: hdfs://rozzanohadoopcluster/hbase/WALs/rzv-db09-hd.rozzano.diennea.lan,16020,1638791923537/rzv-db09-hd.rozzano.diennea.lan%2C16020%2C1638791923537.1638791930213 at position: -1 Recovered source for cluster/machine(s) replicav1: Total replicated edits: 0, current progress:walGroup [db09%2C16020%2C1634401671527]: currently replicating from: hdfs://rozzanohadoopcluster/hbase/oldWALs/rzv-db09-hd.rozzano.diennea.lan%2C16020%2C1634401671527.1634401679218 at position: -1 Recovered source for cluster/machine(s) replicav1: Total replicated edits: 0, current progress:walGroup [db10%2C16020%2C1637585899997]: currently replicating from: hdfs://rozzanohadoopcluster/hbase/oldWALs/rzv-db10-hd.rozzano.diennea.lan%2C16020%2C1637585899997.1637585906625 at position: -1 2021-12-12 08:24:58,561 WARN [regionserver/rzv-db12-hd:16020.logRoller] regionserver.ReplicationSource: WAL group db12%2C16020%2C1638790692057 queue size: 187 exceeds value of replication.source.log.queue.warn: 2 Do you have any info on what could be the problem? Thanks
