my-ship-it commented on issue #90:
URL:
https://github.com/apache/cloudberry-backup/issues/90#issuecomment-4278123838
Hi @woblerr, thanks for raising this — it's a real gap.
`gpbackup_history.db` lives in `$MASTER_DATA_DIRECTORY` but isn't a Postgres
data file, so Cloudberry's built-in standby WAL streaming doesn't cover it.
Today all writes land on the active coordinator only (`backup.go:169` for the
"In Progress" row, `backup.go:519-525` for the final UPDATE in `DoCleanup`), so
a failover really does lose the history. The overall direction — detect
standby, rsync over SSH, non-fatal on failure, opt-out flag — sounds reasonable
to me.
A few points I'd like to discuss before we lock down the design:
**1. Raw rsync of a live SQLite file isn't safe.** SQLite's own docs advise
against file-level copies of an open database (WAL/journal state can produce a
torn snapshot on the destination). Even though `DoCleanup` calls
`historyDB.Close()` before the proposed sync point, a concurrent `gpbackup`
invocation can still be mid-write. Two safer options:
- Export a consistent snapshot first (`VACUUM INTO
'/tmp/gpbackup_history.db.snap'` or the `sqlite3_backup_*` online-backup API),
then rsync the snapshot.
- Or hold the existing history lockfile for the duration of the rsync.
I'd lean toward the snapshot approach — it keeps the sync window short
and doesn't block other backups.
**2. Sync timing — just at the end, or also at start?** The current proposal
only syncs after the final status update. If the process is killed hard (OOM,
host crash) before `DoCleanup` runs, the "In Progress" row never reaches
standby, and post-failover the backup looks like it never happened. Adding a
sync right after the initial insert — debounced or rate-limited if you're
worried about overhead — would close that window.
**3. Standby detection.** Querying `gp_segment_configuration WHERE content =
-1 AND role = 'm' AND status = 'u'` is probably cleaner than shelling out to
`gpstate`. If the standby is marked down, we should skip silently to stay
consistent with the \"non-fatal\" principle.
**4. Destination-side atomicity.** On the standby, write to a temp file
first and \`rename(2)\` into place, so a concurrent reader on the standby
(post-promotion, for example) never observes a half-written DB.
**5. Scope — just the DB?** The \`gpbackup_<ts>_report\` files in MDD are
also useful for post-failover troubleshooting and have the same
single-coordinator problem. Worth deciding up front whether this feature covers
them too, or we leave that for a follow-up.
**6. Flag name.** \`--no-sync-standby\` reads (to me) as if it's about
syncing backup data to the standby in general. Something like
\`--no-history-sync-standby\` or \`--no-standby-history-sync\` would be more
specific. Minor nit, very open to your preference.
**7. SSH assumptions.** Passwordless SSH from coordinator to standby is
standard in Cloudberry clusters, so assuming it is fine — but
containerized/locked-down deployments will fail, which is exactly why your
non-fatal-with-warning default is the right call.
One broader thought: longer term, making the history DB location pluggable
(NFS / object storage, similar to the existing plugin mechanism) would make
this problem go away entirely. Not a reason to block this feature — the standby
sync is a clear incremental win — just flagging it as a possible direction if
we ever revisit.
Happy to help review the PR once you have a draft. Thanks again for the
thoughtful write-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
