Forking: <20220316151253.gb28...@telsasoft.com> On Wed, Mar 16, 2022 at 10:12:54AM -0500, Justin Pryzby wrote: > Also, with a partial regression DB, this crashes when writing to stdout. > > $ src/bin/pg_basebackup/pg_basebackup --wal-method fetch -Ft -D - -h /tmp > --no-sync --compress=lz4 |wc -c > pg_basebackup: bbstreamer_lz4.c:172: bbstreamer_lz4_compressor_content: > Assertion `mystreamer->base.bbs_buffer.maxlen >= out_bound' failed. > 24117248 > > #4 0x000055555555e8b4 in bbstreamer_lz4_compressor_content > (streamer=0x5555555a5260, member=0x7fffffffc760, > data=0x7ffff3068010 "{ \"PostgreSQL-Backup-Manifest-Version\": > 1,\n\"Files\": [\n{ \"Path\": \"backup_label\", \"Size\": 227, > \"Last-Modified\": \"2022-03-16 02:29:11 GMT\", \"Checksum-Algorithm\": > \"CRC32C\", \"Checksum\": \"46f69d99\" },\n{ \"Pa"..., len=401072, > context=BBSTREAMER_MEMBER_CONTENTS) at bbstreamer_lz4.c:172 > mystreamer = 0x5555555a5260 > next_in = 0x7ffff3068010 "{ \"PostgreSQL-Backup-Manifest-Version\": > 1,\n\"Files\": [\n{ \"Path\": \"backup_label\", \"Size\": 227, > \"Last-Modified\": \"2022-03-16 02:29:11 GMT\", \"Checksum-Algorithm\": > \"CRC32C\", \"Checksum\": \"46f69d99\" },\n{ \"Pa"... > ... > > (gdb) p mystreamer->base.bbs_buffer.maxlen > $1 = 524288 > (gdb) p (int) LZ4F_compressBound(len, &mystreamer->prefs) > $4 = 524300 > > This is with: liblz4-1:amd64 1.9.2-2ubuntu0.20.04.1
It looks like maybe this code was copied from bbstreamer_lz4_compressor_finalize() which has an Assert rather than expanding the buffer like in bbstreamer_lz4_compressor_new() commit e70c12214b5ba0bc93c083fdb046304a633018ef Author: Justin Pryzby <pryz...@telsasoft.com> Date: Mon Mar 28 23:24:15 2022 -0500 basebackup: fix crash with lz4 + stdout + manifests That's just one known way to trigger this issue. diff --git a/src/bin/pg_basebackup/bbstreamer_lz4.c b/src/bin/pg_basebackup/bbstreamer_lz4.c index 67f841d96a9..8e8352a450c 100644 --- a/src/bin/pg_basebackup/bbstreamer_lz4.c +++ b/src/bin/pg_basebackup/bbstreamer_lz4.c @@ -170,7 +170,11 @@ bbstreamer_lz4_compressor_content(bbstreamer *streamer, * forward the content to next streamer and empty the buffer. */ out_bound = LZ4F_compressBound(len, &mystreamer->prefs); - Assert(mystreamer->base.bbs_buffer.maxlen >= out_bound); + + /* Enlarge buffer if it falls short of compression bound. */ + if (mystreamer->base.bbs_buffer.maxlen < out_bound) + enlargeStringInfo(&mystreamer->base.bbs_buffer, out_bound); + if (avail_out < out_bound) { bbstreamer_content(mystreamer->base.bbs_next, member,