Somebody ran into issues when generating large XML output (upwards of 256 MB) and then sending via a connection with a different client_encoding. This occurs because we pessimistically allocate 4x as much memory as the string needs, and we run into the 1GB palloc limitation. ISTM we can do better now by using huge allocations, as per the preliminary attached patch (which probably needs an updated overflow check rather than have it removed altogether); but at least it is able to process this query, which it wasn't without the patch:
select query_to_xml( 'select a, cash_words(a::text::money) from generate_series(0, 2000000) a', true, false, ''); -- Álvaro Herrera
>From a718f299d72617c366a08735895e0439482ede2f Mon Sep 17 00:00:00 2001 From: Alvaro Herrera <alvhe...@alvh.no-ip.org> Date: Thu, 8 Aug 2019 15:36:53 -0400 Subject: [PATCH v1] Use huge pallocs on encoding conversions --- src/backend/utils/mb/mbutils.c | 26 ++++---------------------- 1 file changed, 4 insertions(+), 22 deletions(-) diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c index bec54bb5cb..e2853fcf6e 100644 --- a/src/backend/utils/mb/mbutils.c +++ b/src/backend/utils/mb/mbutils.c @@ -348,17 +348,8 @@ pg_do_encoding_conversion(unsigned char *src, int len, pg_encoding_to_char(src_encoding), pg_encoding_to_char(dest_encoding)))); - /* - * Allocate space for conversion result, being wary of integer overflow - */ - if ((Size) len >= (MaxAllocSize / (Size) MAX_CONVERSION_GROWTH)) - ereport(ERROR, - (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), - errmsg("out of memory"), - errdetail("String of %d bytes is too long for encoding conversion.", - len))); - - result = palloc(len * MAX_CONVERSION_GROWTH + 1); + result = palloc_extended((Size) len * MAX_CONVERSION_GROWTH + 1, + MCXT_ALLOC_HUGE); OidFunctionCall5(proc, Int32GetDatum(src_encoding), @@ -681,17 +672,8 @@ perform_default_encoding_conversion(const char *src, int len, if (flinfo == NULL) return unconstify(char *, src); - /* - * Allocate space for conversion result, being wary of integer overflow - */ - if ((Size) len >= (MaxAllocSize / (Size) MAX_CONVERSION_GROWTH)) - ereport(ERROR, - (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), - errmsg("out of memory"), - errdetail("String of %d bytes is too long for encoding conversion.", - len))); - - result = palloc(len * MAX_CONVERSION_GROWTH + 1); + result = palloc_extended((Size) len * MAX_CONVERSION_GROWTH + 1, + MCXT_ALLOC_HUGE); FunctionCall5(flinfo, Int32GetDatum(src_encoding), -- 2.17.1