On Thu, 26 May 2022 at 09:03, Japin Li <[email protected]> wrote:
> On Wed, 25 May 2022 at 22:50, Tom Lane <[email protected]> wrote:
>> Japin Li <[email protected]> writes:
>>> Today, I try to use repeat() to generate 1GB text, and it occurs invalid
>>> memory
>>> alloc request size [1]. It is a limit from palloc(), then I try to reduce
>>> it,
>>> it still complains out of memory which comes from enlargeStringInfo() [2].
>>> The
>>> documentation about repect() [3] doesn't mentaion the limitation.
>>
>> It would probably make sense for repeat() to check this explicitly:
>>
>> if (unlikely(pg_mul_s32_overflow(count, slen, &tlen)) ||
>> - unlikely(pg_add_s32_overflow(tlen, VARHDRSZ, &tlen)))
>> + unlikely(pg_add_s32_overflow(tlen, VARHDRSZ, &tlen)) ||
>> + unlikely(!AllocSizeIsValid(tlen)))
>> ereport(ERROR,
>> (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
>> errmsg("requested length too large")));
>>
>
> LGTM. Thanks for your patch!
After some analysis, I found it might not easy to solve this.
For example,
```
postgres=# CREATE TABLE myrepeat AS SELECT repeat('a', 1024 * 1024 * 1024 - 5);
ERROR: invalid memory alloc request size 1073741871
```
Here is the backtrace:
#0 palloc0 (size=1073741871) at
/mnt/workspace/postgresql/build/../src/backend/utils/mmgr/mcxt.c:1103
#1 0x0000561925199faf in heap_form_tuple (tupleDescriptor=0x561927cb4310,
values=0x561927cb4470, isnull=0x561927cb4478)
at
/mnt/workspace/postgresql/build/../src/backend/access/common/heaptuple.c:1069
#2 0x00005619254879aa in tts_virtual_copy_heap_tuple (slot=0x561927cb4428) at
/mnt/workspace/postgresql/build/../src/backend/executor/execTuples.c:280
#3 0x000056192548a40c in ExecFetchSlotHeapTuple (slot=0x561927cb4428,
materialize=true, shouldFree=0x7ffc9cc5197f)
at /mnt/workspace/postgresql/build/../src/backend/executor/execTuples.c:1660
#4 0x000056192520e0c9 in heapam_tuple_insert (relation=0x7fdacaa9d3e0,
slot=0x561927cb4428, cid=5, options=2, bistate=0x561927cb83f0)
at
/mnt/workspace/postgresql/build/../src/backend/access/heap/heapam_handler.c:245
#5 0x00005619253b2edf in table_tuple_insert (rel=0x7fdacaa9d3e0,
slot=0x561927cb4428, cid=5, options=2, bistate=0x561927cb83f0)
at /mnt/workspace/postgresql/build/../src/include/access/tableam.h:1376
#6 0x00005619253b3d73 in intorel_receive (slot=0x561927cb4428,
self=0x561927c82430) at
/mnt/workspace/postgresql/build/../src/backend/commands/createas.c:586
#7 0x0000561925478d86 in ExecutePlan (estate=0x561927cb3ed0,
planstate=0x561927cb4108, use_parallel_mode=false, operation=CMD_SELECT,
sendTuples=true, numberTuples=0,
direction=ForwardScanDirection, dest=0x561927c82430, execute_once=true) at
/mnt/workspace/postgresql/build/../src/backend/executor/execMain.c:1667
#8 0x0000561925476735 in standard_ExecutorRun (queryDesc=0x561927cab990,
direction=ForwardScanDirection, count=0, execute_once=true)
at /mnt/workspace/postgresql/build/../src/backend/executor/execMain.c:363
#9 0x000056192547654b in ExecutorRun (queryDesc=0x561927cab990,
direction=ForwardScanDirection, count=0, execute_once=true)
at /mnt/workspace/postgresql/build/../src/backend/executor/execMain.c:307
#10 0x00005619253b3711 in ExecCreateTableAs (pstate=0x561927c35b00,
stmt=0x561927b5aa70, params=0x0, queryEnv=0x0, qc=0x7ffc9cc52370)
The heap_form_tupe() function need extra 48 bytes for HeapTupleHeaderData and
HEAPTUPLESIZE.
If we use the following, everything is ok.
postgres=# CREATE TABLE myrepeat AS SELECT repeat('a', 1024 * 1024 * 1024 - 5 -
48);
SELECT 1
If we want to send the result to the client. Here is an error.
postgres=# SELECT repeat('a', 1024 * 1024 * 1024 - 5);
ERROR: out of memory
DETAIL: Cannot enlarge string buffer containing 6 bytes by 1073741819 more
bytes.
This is because the printtup() needs send to the number of attributes (2-byte)
and the length of data (4-byte) and then the data (1073741819-byte).
Do those mean we cannot store 1GB to a field [1] and send 1GB of data to the
client?
[1] https://www.postgresql.org/docs/current/limits.html
--
Regrads,
Japin Li.
ChengDu WenWu Information Technology Co.,Ltd.
