Hi, One of our users reported this issue to me on PG18. pg_verifybackup reported success for a zstd-compressed backup, but zstdfailed to decompress it. After debugging with the user’s data, I found that the backup was truncated, but pg_verifybackup does not verify that the final zstd frame completed. I also tested current master, where the problem still exists.
I created a simple repro. First, use the following commands to create a fake
backup folder and truncate one byte from the compressed tar file. The final
zstd command will fail:
```
workdir=$(mktemp -d /tmp/zstd-trunc.XXXXXX)
mkdir "$workdir/truncated"
dd if=/dev/zero of="$workdir/base.tar" bs=1024 count=2
zstd -q -f "$workdir/base.tar" -o "$workdir/base.tar.zst"
size=$(stat -f %z "$workdir/base.tar.zst")
dd if="$workdir/base.tar.zst" of="$workdir/truncated/base.tar.zst" bs=1
count=$((size - 1))
manifest_prefix=$'{"PostgreSQL-Backup-Manifest-Version": 1,\n "Files": [],\n
"WAL-Ranges": [],\n'
printf '%s' "$manifest_prefix" > "$workdir/manifest-prefix"
manifest_checksum=$(shasum -a 256 "$workdir/manifest-prefix" | awk '{print $1}')
printf '%s"Manifest-Checksum": "%s"}\n' "$manifest_prefix" "$manifest_checksum"
> $workdir/truncated/backup_manifest
zstd -t $workdir/truncated/base.tar.zst
```
However, pg_verifybackup doesn't report an error for the truncated backup:
```
% pg_verifybackup -n -F t -s /tmp/zstd-trunc.YHQ9P4/truncated
backup successfully verified
```
Looking at this code in astreamer_zstd_decompressor_content()
```
ret = ZSTD_decompressStream(mystreamer->dctx,
&mystreamer->zstd_outBuf, &inBuf);
if (ZSTD_isError(ret))
pg_fatal("could not decompress data: %s",
ZSTD_getErrorName(ret));
```
While decompressing the truncated tar file, ZSTD_decompressStream() returns 1.
The doc for ZSTD_decompressStream() at [1] says that a return > 0 means that
there is still decoding or flushing to do before the current frame is complete.
In this case, it may indicate an incomplete frame.
So, this patch records the return value in astreamer_zstd_frame and checks it
in astreamer_zstd_decompressor_finalize(). See the attached patch for details.
I also added a loop to call ZSTD_decompressStream() again in
astreamer_zstd_decompressor_finalize(), because the doc [1] says that, when
output.pos == output.size and the return value is greater than 0, the caller
must call ZSTD_decompressStream() again to flush the remaining buffered output.
With the fix, now pg_verifybackup reports a failure:
```
% pg_verifybackup -n -F t -s /tmp/zstd-trunc.YHQ9P4/truncated
pg_verifybackup: error: could not decompress data: compressed stream is
incomplete
```
BTW, I would also like to bump my previous patch [2]. After changing
fe_utils/astreamer_zstd.c, pg_verifybackup is not rebuilt automatically, which
is inconvenient. I felt the pain again while working on this patch.
[1] https://github.com/facebook/zstd/blob/v1.5.7/lib/zstd.h#L887-L905
[2]
https://www.postgresql.org/message-id/3B062561-C39C-4367-AB1E-F4C27BC6F736%40gmail.com
Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/
v1-0001-Fix-detection-of-truncated-zstd-compressed-backup.patch
Description: Binary data
