[
https://issues.apache.org/jira/browse/PHOENIX-7977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tanuj Khurana resolved PHOENIX-7977.
------------------------------------
Resolution: Fixed
> Stage forwarded replication log files in a per-shard staging subdirectory
> before atomic rename
> ----------------------------------------------------------------------------------------------
>
> Key: PHOENIX-7977
> URL: https://issues.apache.org/jira/browse/PHOENIX-7977
> Project: Phoenix
> Issue Type: Sub-task
> Reporter: Tanuj Khurana
> Assignee: Tanuj Khurana
> Priority: Major
>
> When a fallback (standby) cluster forwards a replication log file to the peer
> cluster, ReplicationLogDiscoveryForwarder.processFile() copied the bytes
> directly onto the file's final replay-eligible name
> <shard>/<ts>_<origin>.plog using FileUtil.copy(..., overwrite=false).
> FileUtil.copy creates the destination at that final name before close()
> completes. The standby replay consumer lists the shard directory on its own
> cadence, and can thereforepick up the still-open, half-written file. When it
> does, it force-recovers the HDFS lease on that file (single-writer model),
> which makes the forwarder's own close() fail with LeaseExpiredException.
> That exception propagates out of processFile() before the STORE_AND_FORWARD →
> SYNC_AND_FORWARD throughput check runs, so the HA group's transition back
> toward SYNC is delayed. The source file also remains in the out-progress
> directory and is retried, but each retry can re-lose the same race.
>
> Fix
> Publish forwarded files atomically instead of exposing them mid-copy:
> 1. Add ReplicationShardDirectoryManager.STAGING_SUB_DIRECTORY_NAME =
> ".staging" and getStagingPath(finalPath), which returns
> <shard>/.staging/<file-name>. The staged file keeps its real .plog name but
> lives one level down in a subdirectory.
> 2. The forwarder copies bytes into the staging path (overwrite=true, to
> reclaim any orphan left by a prior crashed attempt), then does a same-shard,
> same-FileSystem rename(staging, dst) to publish once the file is fully
> written.
> 3. On rename returning false: if dst already exists (a retry raced ahead of
> replay) delete the staging copy and treat it as delivered; otherwise throw so
> the source is retried. Delivery stays at-least-once, which is safe because
> replay is idempotent.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
