Re: Testing autovacuum wraparound (including failsafe)

2024-05-22 Thread Alexander Lakhin
Hello, 30.11.2023 10:35, Masahiko Sawada wrote: I've attached new version patches (0002 and 0003 are unchanged except for the commit message). I'll push them, barring any objections. Pushed. I've discovered that the test 001_emergency_vacuum.pl can fail due to a race condition. I can't

Re: Testing autovacuum wraparound (including failsafe)

2024-02-08 Thread Masahiko Sawada
On Thu, Feb 8, 2024 at 4:06 PM Peter Eisentraut wrote: > > On 08.02.24 05:05, Masahiko Sawada wrote: > > On Thu, Feb 8, 2024 at 3:11 AM Peter Eisentraut > > wrote: > >> > >> The way src/test/modules/xid_wraparound/meson.build is written, it > >> installs the xid_wraparound.so module into

Re: Testing autovacuum wraparound (including failsafe)

2024-02-07 Thread Peter Eisentraut
On 08.02.24 05:05, Masahiko Sawada wrote: On Thu, Feb 8, 2024 at 3:11 AM Peter Eisentraut wrote: The way src/test/modules/xid_wraparound/meson.build is written, it installs the xid_wraparound.so module into production installations. For test modules, a different installation code needs to be

Re: Testing autovacuum wraparound (including failsafe)

2024-02-07 Thread Masahiko Sawada
On Thu, Feb 8, 2024 at 3:11 AM Peter Eisentraut wrote: > > The way src/test/modules/xid_wraparound/meson.build is written, it > installs the xid_wraparound.so module into production installations. > For test modules, a different installation code needs to be used. See > neighboring test modules

Re: Testing autovacuum wraparound (including failsafe)

2024-02-07 Thread Peter Eisentraut
The way src/test/modules/xid_wraparound/meson.build is written, it installs the xid_wraparound.so module into production installations. For test modules, a different installation code needs to be used. See neighboring test modules such as src/test/modules/test_rbtree/meson.build for examples.

Re: Testing autovacuum wraparound (including failsafe)

2023-11-30 Thread Masahiko Sawada
On Thu, Nov 30, 2023 at 4:35 PM Masahiko Sawada wrote: > > On Wed, Nov 29, 2023 at 5:27 AM Masahiko Sawada wrote: > > > > On Tue, Nov 28, 2023 at 7:16 PM Daniel Gustafsson wrote: > > > > > > > On 28 Nov 2023, at 03:00, Masahiko Sawada wrote: > > > > > > > > On Mon, Nov 27, 2023 at 10:40 PM

Re: Testing autovacuum wraparound (including failsafe)

2023-11-29 Thread Masahiko Sawada
On Wed, Nov 29, 2023 at 5:27 AM Masahiko Sawada wrote: > > On Tue, Nov 28, 2023 at 7:16 PM Daniel Gustafsson wrote: > > > > > On 28 Nov 2023, at 03:00, Masahiko Sawada wrote: > > > > > > On Mon, Nov 27, 2023 at 10:40 PM Daniel Gustafsson > > > wrote: > > >> > > >>> On 27 Nov 2023, at 14:06,

Re: Testing autovacuum wraparound (including failsafe)

2023-11-28 Thread Masahiko Sawada
On Tue, Nov 28, 2023 at 7:16 PM Daniel Gustafsson wrote: > > > On 28 Nov 2023, at 03:00, Masahiko Sawada wrote: > > > > On Mon, Nov 27, 2023 at 10:40 PM Daniel Gustafsson wrote: > >> > >>> On 27 Nov 2023, at 14:06, Masahiko Sawada wrote: > >> > >>> Is it true that we can modify the timeout

Re: Testing autovacuum wraparound (including failsafe)

2023-11-28 Thread Daniel Gustafsson
> On 28 Nov 2023, at 03:00, Masahiko Sawada wrote: > > On Mon, Nov 27, 2023 at 10:40 PM Daniel Gustafsson wrote: >> >>> On 27 Nov 2023, at 14:06, Masahiko Sawada wrote: >> >>> Is it true that we can modify the timeout after creating >>> BackgroundPsql object? If so, it seems we don't need to

Re: Testing autovacuum wraparound (including failsafe)

2023-11-27 Thread Masahiko Sawada
On Mon, Nov 27, 2023 at 10:40 PM Daniel Gustafsson wrote: > > > On 27 Nov 2023, at 14:06, Masahiko Sawada wrote: > > > Is it true that we can modify the timeout after creating > > BackgroundPsql object? If so, it seems we don't need to introduce the > > new timeout argument. But how? > > I can't

Re: Testing autovacuum wraparound (including failsafe)

2023-11-27 Thread Daniel Gustafsson
> On 27 Nov 2023, at 14:06, Masahiko Sawada wrote: > Is it true that we can modify the timeout after creating > BackgroundPsql object? If so, it seems we don't need to introduce the > new timeout argument. But how? I can't remember if that's leftovers that incorrectly remains from an earlier

Re: Testing autovacuum wraparound (including failsafe)

2023-11-27 Thread Masahiko Sawada
On Fri, Sep 29, 2023 at 7:17 PM Daniel Gustafsson wrote: > > > On 27 Sep 2023, at 14:39, Masahiko Sawada wrote: > > > I've attached new version patches. 0001 patch adds an option to > > background_psql to specify the timeout seconds, and 0002 patch is the > > main regression test patch. > >

Re: Testing autovacuum wraparound (including failsafe)

2023-09-29 Thread Noah Misch
On Fri, Sep 29, 2023 at 12:17:04PM +0200, Daniel Gustafsson wrote: > +# Bump the query timeout to avoid false negatives on slow test systems. > +my $psql_timeout_secs = 4 * $PostgreSQL::Test::Utils::timeout_default; > > Should we bump the timeout like this for all systems? I interpreted Noah's >

Re: Testing autovacuum wraparound (including failsafe)

2023-09-29 Thread vignesh C
On Thu, 28 Sept 2023 at 03:55, Masahiko Sawada wrote: > > Sorry for the late reply. > > On Sun, Sep 3, 2023 at 2:48 PM Noah Misch wrote: > > > > On Wed, Jul 12, 2023 at 01:47:51PM +0200, Daniel Gustafsson wrote: > > > > On 12 Jul 2023, at 09:52, Masahiko Sawada wrote: > > > > Agreed. The

Re: Testing autovacuum wraparound (including failsafe)

2023-09-29 Thread Daniel Gustafsson
> On 27 Sep 2023, at 14:39, Masahiko Sawada wrote: > I've attached new version patches. 0001 patch adds an option to > background_psql to specify the timeout seconds, and 0002 patch is the > main regression test patch. -=item PostgreSQL::Test::BackgroundPsql->new(interactive, @params) +=item

Re: Testing autovacuum wraparound (including failsafe)

2023-09-27 Thread Masahiko Sawada
Sorry for the late reply. On Sun, Sep 3, 2023 at 2:48 PM Noah Misch wrote: > > On Wed, Jul 12, 2023 at 01:47:51PM +0200, Daniel Gustafsson wrote: > > > On 12 Jul 2023, at 09:52, Masahiko Sawada wrote: > > > Agreed. The timeout can be set by manually setting > > > PG_TEST_TIMEOUT_DEFAULT, but I

Re: Testing autovacuum wraparound (including failsafe)

2023-09-02 Thread Noah Misch
On Wed, Jul 12, 2023 at 01:47:51PM +0200, Daniel Gustafsson wrote: > > On 12 Jul 2023, at 09:52, Masahiko Sawada wrote: > > Agreed. The timeout can be set by manually setting > > PG_TEST_TIMEOUT_DEFAULT, but I bump it to 10 min by default. And it > > now require setting PG_TET_EXTRA to run it. >

Re: Testing autovacuum wraparound (including failsafe)

2023-08-25 Thread Daniel Gustafsson
> On 22 Aug 2023, at 07:49, Michael Paquier wrote: > > On Wed, Jul 12, 2023 at 01:47:51PM +0200, Daniel Gustafsson wrote: >> +# bump the query timeout to avoid false negatives on slow test syetems. >> +$ENV{PG_TEST_TIMEOUT_DEFAULT} = 600; >> Does this actually work? Utils.pm read the

Re: Testing autovacuum wraparound (including failsafe)

2023-08-21 Thread Michael Paquier
On Wed, Jul 12, 2023 at 01:47:51PM +0200, Daniel Gustafsson wrote: > +# bump the query timeout to avoid false negatives on slow test syetems. > +$ENV{PG_TEST_TIMEOUT_DEFAULT} = 600; > Does this actually work? Utils.pm read the environment variable at compile > time in the BEGIN block so this

Re: Testing autovacuum wraparound (including failsafe)

2023-07-12 Thread Daniel Gustafsson
> On 12 Jul 2023, at 09:52, Masahiko Sawada wrote: > Agreed. The timeout can be set by manually setting > PG_TEST_TIMEOUT_DEFAULT, but I bump it to 10 min by default. And it > now require setting PG_TET_EXTRA to run it. +# bump the query timeout to avoid false negatives on slow test syetems.

Re: Testing autovacuum wraparound (including failsafe)

2023-07-12 Thread Masahiko Sawada
On Thu, Apr 27, 2023 at 11:12 PM Daniel Gustafsson wrote: > > > On 27 Apr 2023, at 16:06, Masahiko Sawada wrote: > > On Fri, Apr 21, 2023 at 12:02 PM John Naylor > > wrote: > > >> ...that call to background_psql doesn't look like other ones that have > >> "key => value". Is there something I'm

Re: Testing autovacuum wraparound (including failsafe)

2023-04-28 Thread Daniel Gustafsson
> On 28 Apr 2023, at 06:42, Tom Lane wrote: > John Naylor writes: >> If they're that slow, I'd worry more about generating 20GB of xact status >> data. That's why the tests are disabled by default. > > There is exactly zero chance that anyone will accept the introduction > of such an expensive

Re: Testing autovacuum wraparound (including failsafe)

2023-04-27 Thread Tom Lane
John Naylor writes: > On Thu, Apr 27, 2023 at 9:12 PM Daniel Gustafsson wrote: >> Is there a risk that the background psql will time out on slow systems during >> the consumption of 2B xid's? Since you mainly want to hold it open for the >> duration of testing you might want to bump it to avoid

Re: Testing autovacuum wraparound (including failsafe)

2023-04-27 Thread John Naylor
On Thu, Apr 27, 2023 at 9:12 PM Daniel Gustafsson wrote: > > > On 27 Apr 2023, at 16:06, Masahiko Sawada wrote: > > On Fri, Apr 21, 2023 at 12:02 PM John Naylor > > wrote: > > >> ...that call to background_psql doesn't look like other ones that have "key => value". Is there something I'm

Re: Testing autovacuum wraparound (including failsafe)

2023-04-27 Thread Daniel Gustafsson
> On 27 Apr 2023, at 16:06, Masahiko Sawada wrote: > On Fri, Apr 21, 2023 at 12:02 PM John Naylor > wrote: >> ...that call to background_psql doesn't look like other ones that have "key >> => value". Is there something I'm missing? > > Thanks for reporting. I think that the patch needs to be

Re: Testing autovacuum wraparound (including failsafe)

2023-04-27 Thread Masahiko Sawada
On Fri, Apr 21, 2023 at 12:02 PM John Naylor wrote: > > I agree having the new functions in the tree is useful. I also tried running > the TAP tests in v2, but 001 and 002 fail to run: > > Odd number of elements in hash assignment at [...]/Cluster.pm line 2010. > Can't locate object method

Re: Testing autovacuum wraparound (including failsafe)

2023-04-20 Thread John Naylor
I agree having the new functions in the tree is useful. I also tried running the TAP tests in v2, but 001 and 002 fail to run: Odd number of elements in hash assignment at [...]/Cluster.pm line 2010. Can't locate object method "pump_nb" via package "PostgreSQL::Test::BackgroundPsql" at [...] It

Re: Testing autovacuum wraparound (including failsafe)

2023-03-14 Thread Masahiko Sawada
On Wed, Mar 8, 2023 at 1:52 PM Masahiko Sawada wrote: > > On Fri, Mar 3, 2023 at 8:34 PM Heikki Linnakangas wrote: > > > > On 16/11/2022 06:38, Ian Lawrence Barwick wrote: > > > Thanks for the patch. While reviewing the patch backlog, we have > > > determined that > > > the latest version of

Re: Testing autovacuum wraparound (including failsafe)

2023-03-13 Thread Peter Geoghegan
On Mon, Mar 13, 2023 at 3:25 PM Jacob Champion wrote: > Does https://commitfest.postgresql.org/42/4128/ address that > independently enough? I wasn't aware of that patch. It looks like it does exactly what I was arguing in favor of. So yes. -- Peter Geoghegan

Re: Testing autovacuum wraparound (including failsafe)

2023-03-13 Thread Jacob Champion
On Sat, Mar 11, 2023 at 8:47 PM Peter Geoghegan wrote: > I was joking. But I did have a real point: once we have tests for the > xidStopLimit mechanism, why not take the opportunity to correct the > long standing issue with the documentation advising the use of single > user mode? Does

Re: Testing autovacuum wraparound (including failsafe)

2023-03-11 Thread Peter Geoghegan
On Wed, Mar 8, 2023 at 10:47 PM Michael Paquier wrote: > I may be missing something, but you cannot use directly a "postgres" > command in a TAP test, can you? See 1a9d802, that has fixed a problem > when TAP tests run with a privileged account on Windows. I was joking. But I did have a real

Re: Testing autovacuum wraparound (including failsafe)

2023-03-08 Thread Michael Paquier
On Tue, Mar 07, 2023 at 09:21:00PM -0800, Peter Geoghegan wrote: > Surely your tap test should be using single user mode? Perhaps you > missed the obnoxious HINT, that's part of the WARNING that the test > parses? ;-) I may be missing something, but you cannot use directly a "postgres" command

Re: Testing autovacuum wraparound (including failsafe)

2023-03-07 Thread Peter Geoghegan
On Fri, Mar 3, 2023 at 3:34 AM Heikki Linnakangas wrote: > I took a different approach to consuming the XIDs. Instead of setting > nextXID directly, bypassing GetNewTransactionId(), this patch introduces > a helper function to call GetNewTransactionId() repeatedly. But because > that's slow, it

Re: Testing autovacuum wraparound (including failsafe)

2023-03-07 Thread Masahiko Sawada
On Fri, Mar 3, 2023 at 8:34 PM Heikki Linnakangas wrote: > > On 16/11/2022 06:38, Ian Lawrence Barwick wrote: > > Thanks for the patch. While reviewing the patch backlog, we have determined > > that > > the latest version of this patch was submitted before meson support was > > implemented, so

Re: Testing autovacuum wraparound (including failsafe)

2023-03-03 Thread Heikki Linnakangas
On 03/03/2023 13:34, Heikki Linnakangas wrote: On 16/11/2022 06:38, Ian Lawrence Barwick wrote: Thanks for the patch. While reviewing the patch backlog, we have determined that the latest version of this patch was submitted before meson support was implemented, so it should have a "meson.build"

Re: Testing autovacuum wraparound (including failsafe)

2023-03-03 Thread Heikki Linnakangas
On 16/11/2022 06:38, Ian Lawrence Barwick wrote: Thanks for the patch. While reviewing the patch backlog, we have determined that the latest version of this patch was submitted before meson support was implemented, so it should have a "meson.build" file added for consideration for inclusion in

Re: Testing autovacuum wraparound (including failsafe)

2022-11-15 Thread Ian Lawrence Barwick
2022年6月30日(木) 10:40 Masahiko Sawada : > > Hi, > > On Tue, Feb 1, 2022 at 11:58 AM Masahiko Sawada wrote: > > > > On Fri, Jun 11, 2021 at 10:19 AM Andres Freund wrote: > > > > > > Hi, > > > > > > On 2021-06-10 16:42:01 +0300, Anastasia Lubennikova wrote: > > > > Cool. Thank you for working on

Re: Testing autovacuum wraparound (including failsafe)

2022-06-29 Thread Masahiko Sawada
Hi, On Tue, Feb 1, 2022 at 11:58 AM Masahiko Sawada wrote: > > On Fri, Jun 11, 2021 at 10:19 AM Andres Freund wrote: > > > > Hi, > > > > On 2021-06-10 16:42:01 +0300, Anastasia Lubennikova wrote: > > > Cool. Thank you for working on that! > > > Could you please share a WIP patch for the $subj?

Re: Testing autovacuum wraparound (including failsafe)

2022-01-31 Thread Masahiko Sawada
On Fri, Jun 11, 2021 at 10:19 AM Andres Freund wrote: > > Hi, > > On 2021-06-10 16:42:01 +0300, Anastasia Lubennikova wrote: > > Cool. Thank you for working on that! > > Could you please share a WIP patch for the $subj? I'd be happy to help with > > it. > > I've attached the current WIP state,

Re: Testing autovacuum wraparound (including failsafe)

2021-06-10 Thread Andres Freund
Hi, On 2021-06-10 16:42:01 +0300, Anastasia Lubennikova wrote: > Cool. Thank you for working on that! > Could you please share a WIP patch for the $subj? I'd be happy to help with > it. I've attached the current WIP state, which hasn't evolved much since this message... I put the test in

Re: Testing autovacuum wraparound (including failsafe)

2021-06-10 Thread Anastasia Lubennikova
On Thu, Jun 10, 2021 at 10:52 AM Andres Freund wrote: > > I started to write a test for $Subject, which I think we sorely need. > > Currently my approach is to: > - start a cluster, create a few tables with test data > - acquire SHARE UPDATE EXCLUSIVE in a prepared transaction, to prevent >

Re: Testing autovacuum wraparound (including failsafe)

2021-05-24 Thread Peter Geoghegan
On Tue, May 18, 2021 at 12:10 AM Masahiko Sawada wrote: > Since there is the condition "vacrel->num_index_scans == 0" we could > enter the failsafe mode even if the table is less than 4GB, if we > enter lazy_check_wraparound_failsafe() after executing more than one > index scan. Whereas a vacuum

Re: Testing autovacuum wraparound (including failsafe)

2021-05-18 Thread Masahiko Sawada
On Tue, May 18, 2021 at 2:46 PM Masahiko Sawada wrote: > > On Tue, May 18, 2021 at 2:42 PM Peter Geoghegan wrote: > > > > On Mon, May 17, 2021 at 10:29 PM Masahiko Sawada > > wrote: > > > +1 to fix this. Are you already working on fixing this? If not, I'll > > > post a patch. > > > > I posted

Re: Testing autovacuum wraparound (including failsafe)

2021-05-17 Thread Masahiko Sawada
On Tue, May 18, 2021 at 2:42 PM Peter Geoghegan wrote: > > On Mon, May 17, 2021 at 10:29 PM Masahiko Sawada > wrote: > > +1 to fix this. Are you already working on fixing this? If not, I'll > > post a patch. > > I posted a patch recently (last Thursday my time). Perhaps you can review it? Oh,

Re: Testing autovacuum wraparound (including failsafe)

2021-05-17 Thread Peter Geoghegan
On Mon, May 17, 2021 at 10:29 PM Masahiko Sawada wrote: > +1 to fix this. Are you already working on fixing this? If not, I'll > post a patch. I posted a patch recently (last Thursday my time). Perhaps you can review it? -- Peter Geoghegan

Re: Testing autovacuum wraparound (including failsafe)

2021-05-17 Thread Masahiko Sawada
On Sat, Apr 24, 2021 at 11:16 AM Peter Geoghegan wrote: > > On Fri, Apr 23, 2021 at 5:29 PM Andres Freund wrote: > > On 2021-04-23 16:12:33 -0700, Peter Geoghegan wrote: > > > The only reason that I chose 4GB for FAILSAFE_MIN_PAGES is because the > > > related VACUUM_FSM_EVERY_PAGES constant was

Re: Testing autovacuum wraparound (including failsafe)

2021-05-13 Thread Peter Geoghegan
On Fri, Apr 23, 2021 at 7:56 PM Peter Geoghegan wrote: > I'm convinced -- decoupling the logic from the one-pass-not-two pass > case seems likely to be simpler and more useful. For both the one pass > and two pass/has indexes case. Attached draft patch does it that way. -- Peter Geoghegan

Re: Testing autovacuum wraparound (including failsafe)

2021-04-23 Thread Peter Geoghegan
On Fri, Apr 23, 2021 at 7:53 PM Andres Freund wrote: > I mainly suggested it because to me the current seems hard to > understand. I do think it'd be better to check more often. But checking > depending on the amount of dead tuples at the right time doesn't strike > me as a good idea - a lot of

Re: Testing autovacuum wraparound (including failsafe)

2021-04-23 Thread Andres Freund
Hi, On 2021-04-23 19:42:30 -0700, Peter Geoghegan wrote: > On Fri, Apr 23, 2021 at 7:33 PM Andres Freund wrote: > > Check it every so often, independent of whether there are indexes or > > dead tuples? Or just check it at the boundaries. > > I think that the former suggestion might be better --

Re: Testing autovacuum wraparound (including failsafe)

2021-04-23 Thread Peter Geoghegan
On Fri, Apr 23, 2021 at 7:33 PM Andres Freund wrote: > Check it every so often, independent of whether there are indexes or > dead tuples? Or just check it at the boundaries. I think that the former suggestion might be better -- I actually thought about doing it that way myself. The latter

Re: Testing autovacuum wraparound (including failsafe)

2021-04-23 Thread Andres Freund
Hi, On 2021-04-23 19:15:43 -0700, Peter Geoghegan wrote: > > The failsafe mode affects the table scan itself by disabling cost > > limiting. As far as I can see the ways it triggers for the table scan (vs > > truncation or index processing) are: > > > > 1) Before vacuuming starts, for heap phases

Re: Testing autovacuum wraparound (including failsafe)

2021-04-23 Thread Peter Geoghegan
On Fri, Apr 23, 2021 at 5:29 PM Andres Freund wrote: > On 2021-04-23 16:12:33 -0700, Peter Geoghegan wrote: > > The only reason that I chose 4GB for FAILSAFE_MIN_PAGES is because the > > related VACUUM_FSM_EVERY_PAGES constant was 8GB -- the latter limits > > how often we'll consider the failsafe

Re: Testing autovacuum wraparound (including failsafe)

2021-04-23 Thread Andres Freund
Hi, On 2021-04-23 16:12:33 -0700, Peter Geoghegan wrote: > The only reason that I chose 4GB for FAILSAFE_MIN_PAGES is because the > related VACUUM_FSM_EVERY_PAGES constant was 8GB -- the latter limits > how often we'll consider the failsafe in the single-pass/no-indexes > case. I don't really

Re: Testing autovacuum wraparound (including failsafe)

2021-04-23 Thread Andres Freund
Hi, On 2021-04-23 18:08:12 -0500, Justin Pryzby wrote: > On Fri, Apr 23, 2021 at 01:43:06PM -0700, Andres Freund wrote: > > 2) FAILSAFE_MIN_PAGES is 4GB - which seems to make it infeasible to test the > >failsafe mode, we can't really create 4GB relations on the BF. While > >writing the

Re: Testing autovacuum wraparound (including failsafe)

2021-04-23 Thread Peter Geoghegan
On Fri, Apr 23, 2021 at 1:43 PM Andres Freund wrote: > I started to write a test for $Subject, which I think we sorely need. +1 > Currently my approach is to: > - start a cluster, create a few tables with test data > - acquire SHARE UPDATE EXCLUSIVE in a prepared transaction, to prevent >

Re: Testing autovacuum wraparound (including failsafe)

2021-04-23 Thread Justin Pryzby
On Fri, Apr 23, 2021 at 01:43:06PM -0700, Andres Freund wrote: > 2) FAILSAFE_MIN_PAGES is 4GB - which seems to make it infeasible to test the >failsafe mode, we can't really create 4GB relations on the BF. While >writing the tests I've lowered this to 4MB... > For 2), I don't really have

Testing autovacuum wraparound (including failsafe)

2021-04-23 Thread Andres Freund
Hi, I started to write a test for $Subject, which I think we sorely need. Currently my approach is to: - start a cluster, create a few tables with test data - acquire SHARE UPDATE EXCLUSIVE in a prepared transaction, to prevent autovacuum from doing anything - cause dead tuples to exist -