Hi all, (Andrew in CC.) While reading Andrew's commit 2b5ba2a0a141, I was a bit sad to not see tests for these problems with pglz, applied with the fix down to v14. Relying on fuzzing is not really cool, because these consume resources and they may not even hit the correct target, and we want a maximum of deterministic tests.
And then, I got reminded that one of my pet plugins does something close to that (used that around 9.5 for some FPW compression benchmarks): https://github.com/michaelpq/pg_plugins/tree/main/compress_test With this infrastructure already at hand, implementing the problematic tests with corrupted varlenas was a matter of minutes, leading me to the attached patch (bonus points for check_comprete and rawsize). I would like to apply that down to v14, like the previous commit that has fixed these cases with pglz. That should come in handy in case more bugs pop in this area of the code, especially with more compression methods in mind. Any objections and/or comments about that? -- Michael
From 39eb787808d77d3cad30dc3d5ce2d02503326cba Mon Sep 17 00:00:00 2001 From: Michael Paquier <[email protected]> Date: Mon, 13 Apr 2026 09:33:30 +0900 Subject: [PATCH] test_compression: Test module for compression methods The goal of this module is to provide tests for low-level APIs of compression methods. pglz is covered in this commit. This module includes also tests for the cases detected by fuzzing related to corrupted data, as fixed in 2b5ba2a0a141: - Control byte with match tag bit set, where no data follows. - Control byte with match tag bit set, where 1 byte follows. - Extension byte needed (len=18), where no data follows. As bonus points, tests are added for compress/decompress roundtrips, and for check_complete=false/true. Backpatch-through: 14 --- src/test/modules/Makefile | 1 + src/test/modules/meson.build | 1 + src/test/modules/test_compression/.gitignore | 4 + src/test/modules/test_compression/Makefile | 23 +++++ .../expected/test_compression.out | 51 +++++++++++ src/test/modules/test_compression/meson.build | 33 +++++++ .../test_compression/sql/test_compression.sql | 36 ++++++++ .../test_compression--1.0.sql | 12 +++ .../test_compression/test_compression.c | 85 +++++++++++++++++++ .../test_compression/test_compression.control | 4 + 10 files changed, 250 insertions(+) create mode 100644 src/test/modules/test_compression/.gitignore create mode 100644 src/test/modules/test_compression/Makefile create mode 100644 src/test/modules/test_compression/expected/test_compression.out create mode 100644 src/test/modules/test_compression/meson.build create mode 100644 src/test/modules/test_compression/sql/test_compression.sql create mode 100644 src/test/modules/test_compression/test_compression--1.0.sql create mode 100644 src/test/modules/test_compression/test_compression.c create mode 100644 src/test/modules/test_compression/test_compression.control diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile index 0a74ab5c86f5..dbcd432f8c86 100644 --- a/src/test/modules/Makefile +++ b/src/test/modules/Makefile @@ -22,6 +22,7 @@ SUBDIRS = \ test_bloomfilter \ test_cloexec \ test_checksums \ + test_compression \ test_copy_callbacks \ test_custom_rmgrs \ test_custom_stats \ diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build index 4bca42bb3706..7cb26400d435 100644 --- a/src/test/modules/meson.build +++ b/src/test/modules/meson.build @@ -22,6 +22,7 @@ subdir('test_bitmapset') subdir('test_bloomfilter') subdir('test_cloexec') subdir('test_checksums') +subdir('test_compression') subdir('test_copy_callbacks') subdir('test_cplusplusext') subdir('test_custom_rmgrs') diff --git a/src/test/modules/test_compression/.gitignore b/src/test/modules/test_compression/.gitignore new file mode 100644 index 000000000000..5dcb3ff97235 --- /dev/null +++ b/src/test/modules/test_compression/.gitignore @@ -0,0 +1,4 @@ +# Generated subdirectories +/log/ +/results/ +/tmp_check/ diff --git a/src/test/modules/test_compression/Makefile b/src/test/modules/test_compression/Makefile new file mode 100644 index 000000000000..82c6ace4dc8a --- /dev/null +++ b/src/test/modules/test_compression/Makefile @@ -0,0 +1,23 @@ +# src/test/modules/test_compression/Makefile + +MODULE_big = test_compression +OBJS = \ + $(WIN32RES) \ + test_compression.o +PGFILEDESC = "test_compression - test code for compression methods" + +EXTENSION = test_compression +DATA = test_compression--1.0.sql + +REGRESS = test_compression + +ifdef USE_PGXS +PG_CONFIG = pg_config +PGXS := $(shell $(PG_CONFIG) --pgxs) +include $(PGXS) +else +subdir = src/test/modules/test_compression +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global +include $(top_srcdir)/contrib/contrib-global.mk +endif diff --git a/src/test/modules/test_compression/expected/test_compression.out b/src/test/modules/test_compression/expected/test_compression.out new file mode 100644 index 000000000000..837acc85dd49 --- /dev/null +++ b/src/test/modules/test_compression/expected/test_compression.out @@ -0,0 +1,51 @@ +CREATE EXTENSION test_compression; +-- Round-trip with pglz: compress then decompress. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 400, false) = + decode(repeat('abcd', 100), 'escape') AS roundtrip_ok; + roundtrip_ok +-------------- + t +(1 row) + +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 400, true) = + decode(repeat('abcd', 100), 'escape') AS roundtrip_ok; + roundtrip_ok +-------------- + t +(1 row) + +-- Decompression with rawsize too large, fails to fill the destination +-- buffer. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 500, true); +ERROR: pglz_decompress failed +-- Decompression with rawsize too small, fails with source not fully +-- consumed. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 100, true); +ERROR: pglz_decompress failed +-- Corrupted compressed data. The control byte is set with match tag bit, +-- but only 1 byte follows. +SELECT test_pglz_decompress('\x01ff'::bytea, 1024, false); +ERROR: pglz_decompress failed +SELECT test_pglz_decompress('\x01ff'::bytea, 1024, true); +ERROR: pglz_decompress failed +-- Corrupted compressed data. Control byte with match tag bit set, where +-- no data follows. +SELECT length(test_pglz_decompress('\x01'::bytea, 1024, false)) AS ctrl_only_len; + ctrl_only_len +--------------- + 0 +(1 row) + +SELECT test_pglz_decompress('\x01'::bytea, 1024, true); +ERROR: pglz_decompress failed +-- Corrupted compressed data. The match tag encodes len=18 (aka the +-- extension byte is needed) but there is no data. +SELECT test_pglz_decompress('\x010f01'::bytea, 1024, false); +ERROR: pglz_decompress failed +SELECT test_pglz_decompress('\x010f01'::bytea, 1024, true); +ERROR: pglz_decompress failed +DROP EXTENSION test_compression; diff --git a/src/test/modules/test_compression/meson.build b/src/test/modules/test_compression/meson.build new file mode 100644 index 000000000000..b25144ce71cd --- /dev/null +++ b/src/test/modules/test_compression/meson.build @@ -0,0 +1,33 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +test_compression_sources = files( + 'test_compression.c', +) + +if host_system == 'windows' + test_compression_sources += rc_lib_gen.process(win32ver_rc, extra_args: [ + '--NAME', 'test_compression', + '--FILEDESC', 'test_compression - test code for compression methods',]) +endif + +test_compression = shared_module('test_compression', + test_compression_sources, + kwargs: pg_test_mod_args, +) +test_install_libs += test_compression + +test_install_data += files( + 'test_compression.control', + 'test_compression--1.0.sql', +) + +tests += { + 'name': 'test_compression', + 'sd': meson.current_source_dir(), + 'bd': meson.current_build_dir(), + 'regress': { + 'sql': [ + 'test_compression', + ], + }, +} diff --git a/src/test/modules/test_compression/sql/test_compression.sql b/src/test/modules/test_compression/sql/test_compression.sql new file mode 100644 index 000000000000..4775b5ab582e --- /dev/null +++ b/src/test/modules/test_compression/sql/test_compression.sql @@ -0,0 +1,36 @@ +CREATE EXTENSION test_compression; + +-- Round-trip with pglz: compress then decompress. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 400, false) = + decode(repeat('abcd', 100), 'escape') AS roundtrip_ok; +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 400, true) = + decode(repeat('abcd', 100), 'escape') AS roundtrip_ok; + +-- Decompression with rawsize too large, fails to fill the destination +-- buffer. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 500, true); + +-- Decompression with rawsize too small, fails with source not fully +-- consumed. +SELECT test_pglz_decompress(test_pglz_compress( + decode(repeat('abcd', 100), 'escape')), 100, true); + +-- Corrupted compressed data. The control byte is set with match tag bit, +-- but only 1 byte follows. +SELECT test_pglz_decompress('\x01ff'::bytea, 1024, false); +SELECT test_pglz_decompress('\x01ff'::bytea, 1024, true); + +-- Corrupted compressed data. Control byte with match tag bit set, where +-- no data follows. +SELECT length(test_pglz_decompress('\x01'::bytea, 1024, false)) AS ctrl_only_len; +SELECT test_pglz_decompress('\x01'::bytea, 1024, true); + +-- Corrupted compressed data. The match tag encodes len=18 (aka the +-- extension byte is needed) but there is no data. +SELECT test_pglz_decompress('\x010f01'::bytea, 1024, false); +SELECT test_pglz_decompress('\x010f01'::bytea, 1024, true); + +DROP EXTENSION test_compression; diff --git a/src/test/modules/test_compression/test_compression--1.0.sql b/src/test/modules/test_compression/test_compression--1.0.sql new file mode 100644 index 000000000000..cc789df87340 --- /dev/null +++ b/src/test/modules/test_compression/test_compression--1.0.sql @@ -0,0 +1,12 @@ +/* src/test/modules/test_compression/test_compression--1.0.sql */ + +-- complain if script is sourced in psql, rather than via CREATE EXTENSION +\echo Use "CREATE EXTENSION test_compression" to load this file. \quit + +CREATE FUNCTION test_pglz_compress(bytea) +RETURNS bytea +AS 'MODULE_PATHNAME' LANGUAGE C STRICT; + +CREATE FUNCTION test_pglz_decompress(bytea, int4, bool) +RETURNS bytea +AS 'MODULE_PATHNAME' LANGUAGE C STRICT; diff --git a/src/test/modules/test_compression/test_compression.c b/src/test/modules/test_compression/test_compression.c new file mode 100644 index 000000000000..2a1d27999395 --- /dev/null +++ b/src/test/modules/test_compression/test_compression.c @@ -0,0 +1,85 @@ +/*-------------------------------------------------------------------------- + * + * test_compression.c + * Test harness for compression methods. + * + * Copyright (c) 2026, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/test/modules/test_compression/test_compression.c + * + * ------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "varatt.h" + +PG_MODULE_MAGIC; + +PG_FUNCTION_INFO_V1(test_pglz_compress); +PG_FUNCTION_INFO_V1(test_pglz_decompress); + +/* + * test_pglz_compress + * + * Compress the input using pglz_compress(). Only the "always" strategy is + * currently supported. + * + * Returns the compressed data, or NULL if compression fails. + */ +Datum +test_pglz_compress(PG_FUNCTION_ARGS) +{ + bytea *input = PG_GETARG_BYTEA_PP(0); + char *source = VARDATA_ANY(input); + int32 slen = VARSIZE_ANY_EXHDR(input); + int32 maxout = PGLZ_MAX_OUTPUT(slen); + bytea *result; + int32 clen; + + result = (bytea *) palloc(maxout + VARHDRSZ); + clen = pglz_compress(source, slen, VARDATA(result), + PGLZ_strategy_always); + if (clen < 0) + PG_RETURN_NULL(); + + SET_VARSIZE(result, clen + VARHDRSZ); + PG_RETURN_BYTEA_P(result); +} + +/* + * test_pglz_decompress + * + * Decompress the input using pglz_decompress(). + * + * The second argument is the expected uncompressed data size. The third + * argument is here for the check_complete flag. + * + * Returns the decompressed data, or raises an error if decompression fails. + */ +Datum +test_pglz_decompress(PG_FUNCTION_ARGS) +{ + bytea *input = PG_GETARG_BYTEA_PP(0); + int32 rawsize = PG_GETARG_INT32(1); + bool check_complete = PG_GETARG_BOOL(2); + char *source = VARDATA_ANY(input); + int32 slen = VARSIZE_ANY_EXHDR(input); + bytea *result; + int32 dlen; + + if (rawsize < 0) + elog(ERROR, "rawsize must not be negative"); + + result = (bytea *) palloc(rawsize + VARHDRSZ); + + dlen = pglz_decompress(source, slen, VARDATA(result), + rawsize, check_complete); + if (dlen < 0) + elog(ERROR, "pglz_decompress failed"); + + SET_VARSIZE(result, dlen + VARHDRSZ); + PG_RETURN_BYTEA_P(result); +} diff --git a/src/test/modules/test_compression/test_compression.control b/src/test/modules/test_compression/test_compression.control new file mode 100644 index 000000000000..f707d4dfcf51 --- /dev/null +++ b/src/test/modules/test_compression/test_compression.control @@ -0,0 +1,4 @@ +comment = 'Test code for compression methods' +default_version = '1.0' +module_pathname = '$libdir/test_compression' +relocatable = true -- 2.53.0
signature.asc
Description: PGP signature
