Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

[Ajin Cherian]

On Thu, Jul 9, 2020 at 12:28 PM Amit Kapila <amit.kapil...@gmail.com> wrote:

> On Wed, Jul 8, 2020 at 7:31 PM Ajin Cherian <itsa...@gmail.com> wrote:
>
> Thanks for showing the interest in patch.  How have you ensured that
> streaming is happening?  I don't think the proposed patch can ensure
> it for every case because we also rely on logical_decoding_work_mem to
> decide whether to stream/spill, see ReorderBufferCheckMemoryLimit.  I
> think with your patch it will allow streaming for cases where we have
> large amount of WAL to decode.
>
>
Maybe I missed something but I looked at ReorderBufferCheckMemoryLimit,
even there it checks the same function ReorderBufferCanStream () and
decides whether to stream or spill. Did I miss something?

    while (rb->size >= logical_decoding_work_mem * 1024L)
    {
        /*
         * Pick the largest transaction (or subtransaction) and evict it from
         * memory by streaming, if supported. Otherwise, spill to disk.
         */
        if (ReorderBufferCanStream(rb) &&
            (txn = ReorderBufferLargestTopTXN(rb)) != NULL)
        {
            /* we know there has to be one, because the size is not zero */
            Assert(txn && !txn->toptxn);
            Assert(txn->total_size > 0);
            Assert(rb->size >= txn->total_size);

            ReorderBufferStreamTXN(rb, txn);
        }
        else
        {

I will also add debug and test as you suggested.

regards,
Ajin Cherian
Fujitsu Australia