lastOverflowedXid does not handle transaction ID wraparound

2021-10-12 Thread Stan Hu
In a blog post (https://about.gitlab.com/blog/2021/09/29/why-we-spent-the-last-month-eliminating-postgresql-subtransactions/), I described how PostgreSQL can enter into a suboverflow condition on the replica under a number of conditions: 1. A long transaction starts. 2. A single SAVEPOINT is issu

Re: lastOverflowedXid does not handle transaction ID wraparound

2021-10-17 Thread Dmitry Dolgov
> On Tue, Oct 12, 2021 at 09:53:22PM -0700, Stan Hu wrote: > > I described how PostgreSQL can enter into a suboverflow condition on > the replica under a number of conditions: > > 1. A long transaction starts. > 2. A single SAVEPOINT is issued. > 3. Many rows are updated on the primary, and the sam

Re: lastOverflowedXid does not handle transaction ID wraparound

2021-10-20 Thread Andrey Borodin
> 17 окт. 2021 г., в 21:55, Dmitry Dolgov <9erthali...@gmail.com> написал(а): > I wonder what would be side > effects of clearing it when the snapshot is not suboverfloved anymore? I think we should just invalidate lastOverflowedXid on every XLOG_RUNNING_XACTS if subxid_overflow == false. I ca

Re: lastOverflowedXid does not handle transaction ID wraparound

2021-10-20 Thread Dmitry Dolgov
> On Wed, Oct 20, 2021 at 04:00:35PM +0500, Andrey Borodin wrote: > > 17 окт. 2021 г., в 21:55, Dmitry Dolgov <9erthali...@gmail.com> написал(а): > > I wonder what would be side > > effects of clearing it when the snapshot is not suboverfloved anymore? > > I think we should just invalidate lastOver

Re: lastOverflowedXid does not handle transaction ID wraparound

2021-10-20 Thread Stan Hu
On Wed, Oct 20, 2021 at 4:00 AM Andrey Borodin wrote: > > > > 17 окт. 2021 г., в 21:55, Dmitry Dolgov <9erthali...@gmail.com> > написал(а): > > I wonder what would be side > > effects of clearing it when the snapshot is not suboverfloved anymore? > > I think we should just invalidate lastOverflow

Re: lastOverflowedXid does not handle transaction ID wraparound

2021-10-20 Thread Kyotaro Horiguchi
At Wed, 20 Oct 2021 13:48:33 +0200, Dmitry Dolgov <9erthali...@gmail.com> wrote in > > On Wed, Oct 20, 2021 at 04:00:35PM +0500, Andrey Borodin wrote: > > > 17 окт. 2021 г., в 21:55, Dmitry Dolgov <9erthali...@gmail.com> > > > написал(а): > > > I wonder what would be side > > > effects of cleari

Re: lastOverflowedXid does not handle transaction ID wraparound

2021-10-20 Thread Kyotaro Horiguchi
At Wed, 20 Oct 2021 08:55:12 -0700, Stan Hu wrote in > On Wed, Oct 20, 2021 at 4:00 AM Andrey Borodin wrote: > > > > > > > > 17 окт. 2021 г., в 21:55, Dmitry Dolgov <9erthali...@gmail.com> > > написал(а): > > > I wonder what would be side > > > effects of clearing it when the snapshot is not su

Re: lastOverflowedXid does not handle transaction ID wraparound

2021-10-21 Thread Stan Hu
On Wed, Oct 20, 2021 at 9:01 PM Kyotaro Horiguchi wrote: > > lastOverflowedXid is the smallest subxid that possibly exists but > possiblly not known to the standby. So if all top-level transactions > older than lastOverflowedXid end, that means that all the > subtransactions in doubt are known to

Re: lastOverflowedXid does not handle transaction ID wraparound

2021-10-25 Thread Nikolay Samokhvalov
On Thu, Oct 21, 2021 at 07:21 Stan Hu wrote: > On Wed, Oct 20, 2021 at 9:01 PM Kyotaro Horiguchi > wrote: > > > > lastOverflowedXid is the smallest subxid that possibly exists but > > possiblly not known to the standby. So if all top-level transactions > > older than lastOverflowedXid end, that

Re: lastOverflowedXid does not handle transaction ID wraparound

2021-11-01 Thread Nikolay Samokhvalov
On Mon, Oct 25, 2021 at 11:41 AM Nikolay Samokhvalov wrote: > On Thu, Oct 21, 2021 at 07:21 Stan Hu wrote: > >> On Wed, Oct 20, 2021 at 9:01 PM Kyotaro Horiguchi >> wrote: >> > >> > lastOverflowedXid is the smallest subxid that possibly exists but >> > possiblly not known to the standby. So if

Re: lastOverflowedXid does not handle transaction ID wraparound

2021-11-01 Thread Nikolay Samokhvalov
The following review has been posted through the commitfest application: make installcheck-world: tested, failed Implements feature: tested, failed Spec compliant: not tested Documentation:not tested The fix is trivial and works as expected, solving the problem Tested

Re: lastOverflowedXid does not handle transaction ID wraparound

2021-11-02 Thread Nikolay Samokhvalov
On Mon, Nov 1, 2021 at 11:55 PM Nikolay Samokhvalov wrote: > The following review has been posted through the commitfest application: > make installcheck-world: tested, failed Please ignore this – I didn't understand the UI.

Re: lastOverflowedXid does not handle transaction ID wraparound

2021-11-03 Thread Andrey Borodin
> 21 окт. 2021 г., в 09:01, Kyotaro Horiguchi > написал(а): > > If the discussion so far is correct, the following diff will fix the > issue. > > diff --git a/src/backend/storage/ipc/procarray.c > b/src/backend/storage/ipc/procarray.c > index bd3c7a47fe..19682b73ec 100644 > --- a/src/backen

Re: lastOverflowedXid does not handle transaction ID wraparound

2021-11-03 Thread Alexander Korotkov
( a.On Wed, Nov 3, 2021 at 11:44 AM Andrey Borodin wrote: > > 21 окт. 2021 г., в 09:01, Kyotaro Horiguchi > > написал(а): > > > > If the discussion so far is correct, the following diff will fix the > > issue. > > > > diff --git a/src/backend/storage/ipc/procarray.c > > b/src/backend/storage/ip

Re: lastOverflowedXid does not handle transaction ID wraparound

2021-11-03 Thread Andrey Borodin
> 3 нояб. 2021 г., в 14:08, Alexander Korotkov > написал(а): > > ( a.On Wed, Nov 3, 2021 at 11:44 AM Andrey Borodin > wrote: >>> 21 окт. 2021 г., в 09:01, Kyotaro Horiguchi >>> написал(а): >>> >>> If the discussion so far is correct, the following diff will fix the >>> issue. >>> >>> di

Re: lastOverflowedXid does not handle transaction ID wraparound

2021-11-03 Thread Simon Riggs
On Thu, 21 Oct 2021 at 05:01, Kyotaro Horiguchi wrote: > > At Wed, 20 Oct 2021 08:55:12 -0700, Stan Hu wrote in > > On Wed, Oct 20, 2021 at 4:00 AM Andrey Borodin wrote: > > > > > > > > > > > > 17 окт. 2021 г., в 21:55, Dmitry Dolgov <9erthali...@gmail.com> > > > написал(а): > > > > I wonder wha

Re: lastOverflowedXid does not handle transaction ID wraparound

2021-11-03 Thread Alexander Korotkov
Hi! On Wed, Nov 3, 2021 at 8:51 PM Simon Riggs wrote: > It is however, an undocumented modularity violation. I think that is > acceptable because of the ProcArrayLock traffic, but needs to have a > comment to explain this at the call to > ExpireOldKnownAssignedTransactionIds() i.e. " and potentia

Re: lastOverflowedXid does not handle transaction ID wraparound

2021-11-03 Thread Stan Hu
Good catch on doing this in ExpireAllKnownAssignedTransactionIds() as well. Thanks. Looks good to me! As Nikolay mentioned, I think this is an important bug that we are seeing in production and would appreciate a backport to v12 if possible. On Wed, Nov 3, 2021 at 3:07 PM Alexander Korotkov wrot

Re: lastOverflowedXid does not handle transaction ID wraparound

2021-11-04 Thread Simon Riggs
On Wed, 3 Nov 2021 at 22:07, Alexander Korotkov wrote: > > Hi! > > On Wed, Nov 3, 2021 at 8:51 PM Simon Riggs > wrote: > > It is however, an undocumented modularity violation. I think that is > > acceptable because of the ProcArrayLock traffic, but needs to have a > > comment to explain this at

Re: lastOverflowedXid does not handle transaction ID wraparound

2021-11-05 Thread Kyotaro Horiguchi
At Thu, 4 Nov 2021 01:07:05 +0300, Alexander Korotkov wrote in > Hi! > > On Wed, Nov 3, 2021 at 8:51 PM Simon Riggs > wrote: > > It is however, an undocumented modularity violation. I think that is > > acceptable because of the ProcArrayLock traffic, but needs to have a > > comment to explain

Re: lastOverflowedXid does not handle transaction ID wraparound

2021-11-06 Thread Alexander Korotkov
Hi! On Fri, Nov 5, 2021 at 10:31 AM Kyotaro Horiguchi wrote: > At Thu, 4 Nov 2021 01:07:05 +0300, Alexander Korotkov > wrote in > > On Wed, Nov 3, 2021 at 8:51 PM Simon Riggs > > wrote: > > > It is however, an undocumented modularity violation. I think that is > > > acceptable because of the

Re: lastOverflowedXid does not handle transaction ID wraparound

2021-11-07 Thread Kyotaro Horiguchi
At Sat, 6 Nov 2021 19:16:09 +0300, Alexander Korotkov wrote in > Pushed! Thanks! -- Kyotaro Horiguchi NTT Open Source Software Center