This is an automated email from the ASF dual-hosted git repository.
tuhaihe pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudberry-backup.git
The following commit(s) were added to refs/heads/main by this push:
new 6b40b133 Fix: gprestore --resize-cluster fails with --jobs > 1
6b40b133 is described below
commit 6b40b13354e7ee2ff8f75262892d2f3d6004604e
Author: [email protected] <[email protected]>
AuthorDate: Fri Apr 24 18:16:50 2026 +0800
Fix: gprestore --resize-cluster fails with --jobs > 1
When restoring with --resize-cluster from a larger to smaller segment
count, gprestore expands the oid list into (oid, batch) pairs in
oid-major order ([T1B0, T1B1, T2B0, T2B1, ...]) and preloads
min(NumConns, len(oidList)) segment pipes before starting the helpers.
Coordinator workers take one task per table and iterate batches
sequentially inside restoreSingleTableData, so the k-th worker's first
COPY targets oidList index k*batches. When batches >= 2 and --jobs > 1
(NumConns > 1), workers beyond NumConns/batches race ahead of helper
pipe creation and fail with:
ERROR: command error message: cat: .../<pipe>_<oid>_0: No such file
or directory
and the restore only succeeds with --jobs 1. This matches the known
upstream gpbackup issue fixed in 1.30.6.
Preload NumConns*batches pipes (clamped to len(oidList)) so every
concurrent worker's first batch is covered. The helper then rolls its
queue forward one pipe per completed batch, and since a worker must
finish all batches of its current table before pulling the next task,
helper progress stays ahead of worker demand. For batches == 1 the
behavior is unchanged.
Fixes #92
---
restore/data.go | 18 +++++++++++-------
1 file changed, 11 insertions(+), 7 deletions(-)
diff --git a/restore/data.go b/restore/data.go
index 4175ad96..44d21120 100644
--- a/restore/data.go
+++ b/restore/data.go
@@ -242,7 +242,7 @@ func restoreDataFromTimestamp(fpInfo filepath.FilePathInfo,
dataEntries []toc.Co
}
utils.WriteOidListToSegments(oidList, globalCluster, fpInfo,
"oid")
- initialPipes := CreateInitialSegmentPipes(oidList,
globalCluster, connectionPool, fpInfo)
+ initialPipes := CreateInitialSegmentPipes(oidList,
globalCluster, connectionPool, fpInfo, batches)
if wasTerminated {
return 0
}
@@ -339,12 +339,16 @@ func restoreDataFromTimestamp(fpInfo
filepath.FilePathInfo, dataEntries []toc.Co
return numErrors
}
-func CreateInitialSegmentPipes(oidList []string, c *cluster.Cluster,
connectionPool *dbconn.DBConn, fpInfo filepath.FilePathInfo) int {
- // Create min(connections, tables) segment pipes on each host
- var maxPipes int
- if connectionPool.NumConns < len(oidList) {
- maxPipes = connectionPool.NumConns
- } else {
+func CreateInitialSegmentPipes(oidList []string, c *cluster.Cluster,
connectionPool *dbconn.DBConn, fpInfo filepath.FilePathInfo, batches int) int {
+ // oidList is laid out in oid-major order: [T1B0, T1B1, ..., T2B0,
T2B1, ...].
+ // Workers dispatch one task per table and iterate batches sequentially
within
+ // restoreSingleTableData, so NumConns concurrent workers may request
pipes at
+ // oidList indices 0, batches, 2*batches, ..., (NumConns-1)*batches
before the
+ // helper has had a chance to create any of them. Preload
NumConns*batches
+ // pipes so every concurrent worker's first batch is covered; the
helper then
+ // rolls the queue forward as each batch completes.
+ maxPipes := connectionPool.NumConns * batches
+ if maxPipes > len(oidList) {
maxPipes = len(oidList)
}
for i := 0; i < maxPipes; i++ {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
