On Sun, Oct 4, 2026 at 2:31 PM Dilip Kumar <[email protected]> wrote: > > > And without fix shared as a v77-0003 [1], this would error out with > below error[2], I have explained the reason [1]. I have added a > testcase in 035_conflicts.pl file in v79-0003 patch for this issue. >
Thanks for the patches. I have resumed reviewing this thread. I think
the size cap added in v79-0002 does not fully prevent the >1GB json
value it is meant to guard against.
In build_index_key_json(), the cap is checked against the stored size
(rawsize), but what actually gets appended is the type's output text,
escaped by escape_json(). In some cases, the output can be far larger
than the storage, and the ratio is not the small factor the comment
atop CONFLICT_MAX_VALUE_SIZE assumes. I have attached a test case to
reproduce the problem. In the test, we use composite types such as:
CREATE TYPE ri_n0 AS (f text);
then
CREATE TYPE ri_n1 AS (f ri_n0);
CREATE TYPE ri_n2 AS (f ri_n1);
... going up to ri_n25.
ri_n25 is then used as the key column of a table.
Then we insert a row whose replica identity is
ROW(...ROW(repeat('"',10))...)::ri_n25. On disk it is stored in this
compact nested form (not the expanded one, ~620 bytes), but in
build_index_key_json(), OidOutputFunctionCall (i.e. record_out)
recurses through all 25 nested levels, to expand the value into its
text form, which grows beyond 800MB, and escaping it then exceeds 1GB.
So durng escape_json() post OidOutputFunctionCall(), apply worker
errors out with:
ERROR: string buffer exceeds maximum allowed length (1073741823 bytes)
I have attached the complete steps to reproduce the issue, as well as
a patch to fix the problem. Please review.
thanks
Shveta
On Pub and Sub:
------------------
DO $$
BEGIN
EXECUTE 'CREATE TYPE ri_n0 AS (f text)';
FOR i IN 1..25 LOOP -- levels knob
EXECUTE format('CREATE TYPE ri_n%s AS (f ri_n%s)', i, i - 1);
END LOOP;
END $$;
CREATE TABLE t (k ri_n25 PRIMARY KEY);
Pub:
-----
CREATE PUBLICATION pub1 FOR ALL TABLES;
Sub:
-----
(Use destination as 'table'; 'all'/'log' may end up consuming too much memory
to dump such a huge RI, and thus the CLT-related problem may be hidden.)
CREATE SUBSCRIPTION sub1 CONNECTION '...' PUBLICATION pub1
WITH(CONFLICT_LOG_DESTINATION='table');
Pub:
-----
DO $$
DECLARE r text := format('ROW(%L)::ri_n0', repeat('"', 10));
BEGIN
FOR i IN 1..25 LOOP
r := format('ROW(%s)::ri_n%s', r, i);
END LOOP;
EXECUTE format('INSERT INTO t VALUES (%s)', r);
END $$;
Sub:
-----
--wait for data to be replicated, check stored length:
SELECT pg_column_size(k) FROM t;
Sub:
-----
DELETE FROM t;
Pub (to reproduce delete_missing conflict):
-----
DELETE FROM t; -- subscriber now errors and retries in a loop
0001-Fix-Ri-output-overflow-bug.patch
Description: Binary data
