[
https://issues.apache.org/jira/browse/HBASE-30090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18074944#comment-18074944
]
Kevin Geiszler commented on HBASE-30090:
----------------------------------------
I found this in the region server log on the replica cluster:
{code:java}
2026-04-17T23:27:05,352 ERROR [ProcedureDispatcherTimeoutThread]
procedure2.RemoteProcedureDispatcher: DelayQueue for RemoteProcedureDispatcher
is not empty when timed waiting elapsed. If this is repeated consistently, it
means no element is getting expired from the queue and it might freeze the
system. Queue: [containedObject=hbase-docker-2,16020,1776468295004,
timeout=1776468425396, delay=44, operations=[pid=10, ppid=9, state=RUNNABLE,
hasLock=false; org.apache.hadoop.hbase.master.procedure.FlushRegionProcedure]]
{code}
> Table on replica cluster not refreshing after flipping read-only flag twice
> ---------------------------------------------------------------------------
>
> Key: HBASE-30090
> URL: https://issues.apache.org/jira/browse/HBASE-30090
> Project: HBase
> Issue Type: Sub-task
> Components: read replicas
> Reporter: Kevin Geiszler
> Priority: Major
> Attachments: original-active-cluster-logs.tar.gz,
> original-replica-cluster-logs.tar.gz
>
>
> I have noticed that an existing table on a read-replica cluster is not
> getting updated after making this cluster the active cluster and then making
> it read-only again. If you have an active cluster, create a table, give it
> data, and flush the table, then the replica will see this after refreshing
> meta and hfiles. If you flip the read-only flag on both clusters, and add
> more data, then the new replica will see this data. However, after flipping
> the flag back for both clusters and adding data on the active cluster, the
> replica cluster will not see this new data even after flushing the table on
> the active cluster and refreshing meta and hfiles on the replica. Here are
> some example steps to reproduce this issue on the hbase-docker setup:
> Assume Cluster A starts as the active cluster and Cluster B starts as the
> read-replica cluster.
> # On Cluster A, create a table: create 't1', 'cf'
> # On Cluster A, add data: put 't1', 'r1', 'cf:c1', '1'
> # On Cluster A, flush: flush 't1'
> # On Cluster B, run refresh_meta and refresh_hfiles. Verify 't1' is there
> and has one row.
> # Make Cluster A read-only.
> # Make Cluster B the active cluster.
> # On Cluster B, add a row to 't1': put 't1', 'r2', 'cf:c1', '2'
> # On Cluster B, flush: flush 't1'
> # On Cluster A, run refresh_meta and refresh_hfiles. Verify 't1' is there
> and has two rows.
> # Make Cluster B read-only again.
> # Make Cluster A the active cluster again.
> # On Cluster A, add data to 't1': put 't1', 'r3', 'cf:c1', '3'
> # On Cluster A, flush: flush 't1'
> # On Cluster B, run refresh_meta and refresh_hfiles. The row count for 't1'
> will still be 2 rows rather than 3 rows.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
