回复:回复:another autovacuum scheduling thread

2025-11-13 Thread 段坤仁(刻韧)
rt Haas; "pgsql-hackers" 主 题:Re: 回复:another autovacuum scheduling thread On Fri, Nov 14, 2025 at 02:37:31AM +0800, 段坤仁(刻韧) wrote: > Hi all, I have read the discussion in this thread and am pleased to see the > community working collaboratively to address some long-standing autov

Re: another autovacuum scheduling thread

2025-11-12 Thread Sami Imseih
> > > On Nov 12, 2025, at 5:10 PM, Sami Imseih wrote: > > > >  > >> > >> I do think re-prioritization is worth considering, but IMHO we should > leave > >> it out of phase 1. I think it's pretty easy to reason about one round > of > >> prioritization being okay. The order is completely arbitrar

Re: another autovacuum scheduling thread

2025-11-12 Thread Jeremy Schneider
> On Nov 12, 2025, at 5:10 PM, Sami Imseih wrote: > >  >> >> I do think re-prioritization is worth considering, but IMHO we should leave >> it out of phase 1. I think it's pretty easy to reason about one round of >> prioritization being okay. The order is completely arbitrary today, so how

Re: another autovacuum scheduling thread

2025-11-12 Thread Sami Imseih
> I do think re-prioritization is worth considering, but IMHO we should leave > it out of phase 1. I think it's pretty easy to reason about one round of > prioritization being okay. The order is completely arbitrary today, so how > could ordering by vacuum-related criteria make things any worse?

Re: another autovacuum scheduling thread

2025-11-12 Thread Nathan Bossart
On Tue, Nov 11, 2025 at 06:22:36PM -0500, Robert Treat wrote: > On Tue, Nov 11, 2025 at 3:27 PM David Rowley wrote: >> On Wed, 12 Nov 2025 at 09:13, Nathan Bossart >> wrote: >> > My concern is that this might add already-processed tables back to the >> > list, so a worker might never be able to

Re: another autovacuum scheduling thread

2025-11-11 Thread Robert Treat
On Tue, Nov 11, 2025 at 3:27 PM David Rowley wrote: > On Wed, 12 Nov 2025 at 09:13, Nathan Bossart wrote: > > On Wed, Nov 12, 2025 at 09:03:54AM +1300, David Rowley wrote: > > > /* when enough time has passed, refresh the list to ensure the > > > scores aren't too out-of-date */ > > > if

Re: another autovacuum scheduling thread

2025-11-11 Thread Sami Imseih
> > The thing I’m hoping to address is something I’ve seen many times in > > practice. > > Autovacuum workers can get stuck on specific large or slow tables, and when > > that happens, users often end up running manual vacuums on those tables > > just to keep things moving for the smaller/faster v

Re: another autovacuum scheduling thread

2025-11-11 Thread David Rowley
On Wed, 12 Nov 2025 at 09:25, Sami Imseih wrote: > The thing I’m hoping to address is something I’ve seen many times in practice. > Autovacuum workers can get stuck on specific large or slow tables, and when > that happens, users often end up running manual vacuums on those tables > just to keep t

Re: another autovacuum scheduling thread

2025-11-11 Thread David Rowley
On Wed, 12 Nov 2025 at 09:13, Nathan Bossart wrote: > > On Wed, Nov 12, 2025 at 09:03:54AM +1300, David Rowley wrote: > > /* when enough time has passed, refresh the list to ensure the > > scores aren't too out-of-date */ > > if (time is > lastcheck + autovacuum_naptime * ) > > { > >

Re: another autovacuum scheduling thread

2025-11-11 Thread Sami Imseih
> On Sat, 8 Nov 2025 at 08:23, Sami Imseih wrote: > > > I'm confused at why we'd have set up our autovacuum trigger points as > > > they are today because we think those are good times to do a > > > vacuum/analyze, but then prioritise on something completely different. > > > Surely if we think 20%

Re: another autovacuum scheduling thread

2025-11-11 Thread Nathan Bossart
On Tue, Nov 11, 2025 at 02:50:55PM -0500, Robert Treat wrote: > On Tue, Nov 11, 2025 at 2:49 PM Nathan Bossart > wrote: >> On Tue, Nov 11, 2025 at 02:43:19PM -0500, Robert Treat wrote: >> > FWIW, when I have built these types of systems in the past, and when I >> > wanted an aggressive recheck-ty

Re: another autovacuum scheduling thread

2025-11-11 Thread Nathan Bossart
On Wed, Nov 12, 2025 at 09:03:54AM +1300, David Rowley wrote: > I'm still trying to work out what Sami sees in the results that he > doesn't think is good. I resuggested he try coding up the periodic > refresh-the-list code to see if it makes the thing he sees better. I > was hoping that we could g

Re: another autovacuum scheduling thread

2025-11-11 Thread David Rowley
On Wed, 12 Nov 2025 at 05:36, Nathan Bossart wrote: > From skimming the latest discussion, I gather we might want to consider > re-sorting the list periodically. Is the idea that we'll re-sort the > remaining tables in the list, or that we'll basically restart > do_autovacuum()? If it's the latt

Re: another autovacuum scheduling thread

2025-11-11 Thread Robert Treat
On Tue, Nov 11, 2025 at 2:49 PM Nathan Bossart wrote: > On Tue, Nov 11, 2025 at 02:43:19PM -0500, Robert Treat wrote: > > FWIW, when I have built these types of systems in the past, and when I > > wanted an aggressive recheck-type mechanism, the most common methods > > involved tying it to autovac

Re: another autovacuum scheduling thread

2025-11-11 Thread Nathan Bossart
On Tue, Nov 11, 2025 at 02:43:19PM -0500, Robert Treat wrote: > FWIW, when I have built these types of systems in the past, and when I > wanted an aggressive recheck-type mechanism, the most common methods > involved tying it to autovacuum_max_workers. Would you mind elaborating on this point? Do

Re: another autovacuum scheduling thread

2025-11-11 Thread Robert Treat
On Tue, Nov 11, 2025 at 11:36 AM Nathan Bossart wrote: > > Still catching up on the latest discussion, but here's a v8 patch that > amends the DEBUG3 in relation_needs_vacanalyze() to also log the score. I > might attempt to add some sort of brief documentation about autovacuum > prioritization n

Re: another autovacuum scheduling thread

2025-11-11 Thread Nathan Bossart
Still catching up on the latest discussion, but here's a v8 patch that amends the DEBUG3 in relation_needs_vacanalyze() to also log the score. I might attempt to add some sort of brief documentation about autovacuum prioritization next. >From skimming the latest discussion, I gather we might want

Re: another autovacuum scheduling thread

2025-11-10 Thread David Rowley
On Sat, 8 Nov 2025 at 08:23, Sami Imseih wrote: > > I'm confused at why we'd have set up our autovacuum trigger points as > > they are today because we think those are good times to do a > > vacuum/analyze, but then prioritise on something completely different. > > Surely if we think 20% dead tupl

Re: another autovacuum scheduling thread

2025-11-07 Thread Sami Imseih
> On Fri, 7 Nov 2025 at 11:21, Sami Imseih wrote: > > Also, I am thinking about another sorting strategy based on average > > autovacuum/autoanalyze time per table. The idea is to sort ascending by > > the greater of the two averages, so workers process quicker tables first > > instead of all work

Re: another autovacuum scheduling thread

2025-11-06 Thread David Rowley
On Fri, 7 Nov 2025 at 11:21, Sami Imseih wrote: > Also, I am thinking about another sorting strategy based on average > autovacuum/autoanalyze time per table. The idea is to sort ascending by > the greater of the two averages, so workers process quicker tables first > instead of all workers potent

Re: another autovacuum scheduling thread

2025-11-06 Thread Sami Imseih
Thanks for the ideas on improving the test! I am still trying to see how useful this type of testing is, but I will share what I have done. > I wonder if it would be more realistic to throttle the work simulation > to a certain speed with pgbench -R rather than having it go flat out. good point

Re: another autovacuum scheduling thread

2025-10-31 Thread David Rowley
On Sat, 1 Nov 2025 at 14:50, David Rowley wrote: > If we logged the score, we could do the "unpatched" test with the > patched code, just with commenting out the > list_sort(tables_to_process, TableToProcessComparator); It'd then be > interesting to zero the log_auto*_min_duration settings and rev

Re: another autovacuum scheduling thread

2025-10-31 Thread David Rowley
On Sat, 1 Nov 2025 at 09:12, Nathan Bossart wrote: > > On Thu, Oct 30, 2025 at 07:38:15PM -0500, Sami Imseih wrote: > > The results above show what I expected: the batch tables receive higher > > priority, as seen from the averages of autovacuum and autoanalyze runs. > > This behavior is expected,

Re: another autovacuum scheduling thread

2025-10-31 Thread Nathan Bossart
On Thu, Oct 30, 2025 at 07:38:15PM -0500, Sami Imseih wrote: > Here is my attempt to test the behavior with the new prioritization. Thanks. > The results above show what I expected: the batch tables receive higher > priority, as seen from the averages of autovacuum and autoanalyze runs. > This be

Re: another autovacuum scheduling thread

2025-10-30 Thread Sami Imseih
> FWIW, I've been putting some scripts together to test some workloads > and I will share shortly what I have. Here is my attempt to test the behavior with the new prioritization. I wanted a way to run the same tests with different workloads, both with and without the prioritization patch, and to

Re: another autovacuum scheduling thread

2025-10-30 Thread Sami Imseih
> On Thu, Oct 30, 2025 at 04:05:19PM -0400, Robert Haas wrote: > > Are you planning to do some practical experimentation with this? I > > feel like it would be a good idea to set up some kind of a test case > > where this is expected to provide a benefit and see if it actually > > does; and also ma

Re: another autovacuum scheduling thread

2025-10-30 Thread Nathan Bossart
On Thu, Oct 30, 2025 at 04:05:19PM -0400, Robert Haas wrote: > Are you planning to do some practical experimentation with this? I > feel like it would be a good idea to set up some kind of a test case > where this is expected to provide a benefit and see if it actually > does; and also maybe set up

Re: another autovacuum scheduling thread

2025-10-30 Thread Robert Haas
On Wed, Oct 29, 2025 at 11:51 AM Nathan Bossart wrote: > Oops. I fixed that typo in v7. Are you planning to do some practical experimentation with this? I feel like it would be a good idea to set up some kind of a test case where this is expected to provide a benefit and see if it actually does;

Re: another autovacuum scheduling thread

2025-10-30 Thread David Rowley
On Thu, 30 Oct 2025 at 19:48, wenhui qiu wrote: > I think there might be some misunderstanding — I’m only suggesting > changing > effective_xid_failsafe_age = Max(vacuum_failsafe_age, > autovacuum_freeze_max_age * 1.05); > to > effective_xid_failsafe_age = (v

Re: another autovacuum scheduling thread

2025-10-29 Thread wenhui qiu
HI I think there might be some misunderstanding — I’m only suggesting changing effective_xid_failsafe_age = Max(vacuum_failsafe_age, autovacuum_freeze_max_age * 1.05); to effective_xid_failsafe_age = (vacuum_failsafe_age + autovacuum_freeze_max_age) / 2.0; In t

Re: another autovacuum scheduling thread

2025-10-29 Thread David Rowley
On Thu, 30 Oct 2025 at 15:58, wenhui qiu wrote: > In fact, with the introduction of the vacuum_max_eager_freeze_failure_rate > feature, if a table’s age still exceeds more than 1.x times the > autovacuum_freeze_max_age, it suggests that the vacuum freeze process is not > functioning properly. O

Re: another autovacuum scheduling thread

2025-10-29 Thread wenhui qiu
HI Nathan > That approach would begin aggressively scaling the priority of tables > sooner, but I don't know if that's strictly better. In any case, I'd like > to avoid making the score calculation too magical. In fact, with the introduction of the vacuum_max_eager_freeze_failure_rate feature, if

Re: another autovacuum scheduling thread

2025-10-29 Thread Nathan Bossart
On Wed, Oct 29, 2025 at 10:24:17AM -0500, Sami Imseih wrote: > I think we do need some documentation about this behavior, which v6 is > still missing. Would you be interested in giving that part a try? > Another thing I have been contemplating about is the change in prioritization > and the resul

Re: another autovacuum scheduling thread

2025-10-29 Thread Nathan Bossart
On Wed, Oct 29, 2025 at 11:10:55AM +0800, wenhui qiu wrote: > Typically, DBAs avoid setting autovacuum_freeze_max_age too close to > vacuum_failsafe_age. Therefore, your logic most likely uses the > vacuum_failsafe_age value. > Would taking the average of the two be a better approach? That approac

Re: another autovacuum scheduling thread

2025-10-29 Thread Nathan Bossart
On Tue, Oct 28, 2025 at 05:44:37PM -0500, Sami Imseih wrote: > My compiler is complaining about v6 > > "../src/backend/postmaster/autovacuum.c:3293:32: warning: operation on > ‘*score’ may be undefined [-Wsequence-point] > 3293 | *score = *score = Max(*score, (double) > in

Re: another autovacuum scheduling thread

2025-10-29 Thread Sami Imseih
> On Tue, Oct 28, 2025 at 12:16:28PM +1300, David Rowley wrote: > > I think it's reasonable to want to document how autovacuum prioritises > > tables, but maybe not in too much detail. Longer term, I think it > > would be good to have a pg_catalog view for this which showed the > > relid or schema/

Re: another autovacuum scheduling thread

2025-10-28 Thread wenhui qiu
HI Nathan Bossart > + if (vactuples > vacthresh) > + { > + *dovacuum = true; > + *score = Max(*score, (double) vactuples / Max(vacthresh, 1)); > + } > + > + if (vac_ins_base_thresh >= 0 && instuples > vacinsthresh) > + { > + *dovacuum = true; > + *score = *score = Max(*score, (double) instuples /

Re: another autovacuum scheduling thread

2025-10-28 Thread Sami Imseih
> Done. My compiler is complaining about v6 "../src/backend/postmaster/autovacuum.c:3293:32: warning: operation on ‘*score’ may be undefined [-Wsequence-point] 3293 | *score = *score = Max(*score, (double) instuples / Max(vacinsthresh, 1)); [2/2] Linking target src/backen

Re: another autovacuum scheduling thread

2025-10-28 Thread Nathan Bossart
On Tue, Oct 28, 2025 at 12:16:28PM +1300, David Rowley wrote: > I think it's reasonable to want to document how autovacuum prioritises > tables, but maybe not in too much detail. Longer term, I think it > would be good to have a pg_catalog view for this which showed the > relid or schema/relname, a

Re: another autovacuum scheduling thread

2025-10-28 Thread Nathan Bossart
On Tue, Oct 28, 2025 at 11:47:08AM +1300, David Rowley wrote: > 1. I think the following code at the bottom of > relation_needs_vacanalyze() can be deleted. You've added the check to > ensure *doanalyze never gets set to true for pg_statistic. > > /* ANALYZE refuses to work with pg_statistic */ >

Re: another autovacuum scheduling thread

2025-10-27 Thread David Rowley
On Tue, 28 Oct 2025 at 11:35, Sami Imseih wrote: > We discuss the threshold calculations in the documentation, and users > can write scripts to monitor which tables are eligible. However, there > is nothing that indicates which table autovacuum will work on next (I > have been asked that question

Re: another autovacuum scheduling thread

2025-10-27 Thread David Rowley
The patch is starting to look good. Here's a review of v5: 1. I think the following code at the bottom of relation_needs_vacanalyze() can be deleted. You've added the check to ensure *doanalyze never gets set to true for pg_statistic. /* ANALYZE refuses to work with pg_statistic */ if (relid == S

Re: another autovacuum scheduling thread

2025-10-27 Thread Sami Imseih
> > I wrote a sql that returns the tables and scores, which I found was > > useful when I was testing this out, so having the actually rules spelled out > > in docs will actually be super useful. > > Can you elaborate on how it would be useful? I'd be open to adding a short > note that autovacuum

Re: another autovacuum scheduling thread

2025-10-27 Thread Nathan Bossart
On Mon, Oct 27, 2025 at 12:47:15PM -0500, Sami Imseih wrote: > 1/ Should we add documentation explaining this prioritization behavior in [0]? > > I wrote a sql that returns the tables and scores, which I found was > useful when I was testing this out, so having the actually rules spelled out > in

Re: another autovacuum scheduling thread

2025-10-27 Thread Sami Imseih
I spent some time looking at this, and I am not sure how much this will move the goalpost, since most of the time the bottleneck for autovacuum is the limited number of workers and large tables that take a long time to process. That said, this is a good change for the simple reason that it is bett

Re: another autovacuum scheduling thread

2025-10-27 Thread Nathan Bossart
On Sun, Oct 26, 2025 at 02:25:48PM +1300, David Rowley wrote: > Thanks. I've just had a look at it. A few comments and questions. Thanks. > 1) The subtraction here looks back to front: > > + xid_age = TransactionIdIsNormal(relfrozenxid) ? relfrozenxid - recentXid : > 0; > + mxid_age = MultiXact

Re: another autovacuum scheduling thread

2025-10-25 Thread David Rowley
On Sat, 25 Oct 2025 at 04:08, Nathan Bossart wrote: > Here is an updated patch based on the latest discussion. Thanks. I've just had a look at it. A few comments and questions. 1) The subtraction here looks back to front: + xid_age = TransactionIdIsNormal(relfrozenxid) ? relfrozenxid - recentXi

Re: another autovacuum scheduling thread

2025-10-24 Thread David Rowley
On Sat, 25 Oct 2025 at 10:14, Peter Geoghegan wrote: > > On Wed, Oct 22, 2025 at 3:35 PM David Rowley wrote: > > If we had the varying sleep time as I mentioned above, the > > failsafe code could even be removed as the > > "autovacuum_vacuum_cost_delay / " calculation would > > effectively zero t

Re: another autovacuum scheduling thread

2025-10-24 Thread Peter Geoghegan
On Wed, Oct 22, 2025 at 3:35 PM David Rowley wrote: > If we had the varying sleep time as I mentioned above, the > failsafe code could even be removed as the > "autovacuum_vacuum_cost_delay / " calculation would > effectively zero the sleep time with any table > failsafe age. I'm not sure what yo

Re: another autovacuum scheduling thread

2025-10-24 Thread Nathan Bossart
Here is an updated patch based on the latest discussion. -- nathan >From 232bc8a73711c743e74d964abfb202563e176aa5 Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Fri, 10 Oct 2025 12:28:37 -0500 Subject: [PATCH v4 1/1] autovacuum scheduling improvements --- src/backend/postmaster/autovacuum

Re: another autovacuum scheduling thread

2025-10-23 Thread David Rowley
On Fri, 24 Oct 2025 at 09:48, Sami Imseih wrote: > Yes, in my last reply, I did indicate that the sort will likely not be > the operation that will tip the performance over, but the > catalog scan itself that I have seen not scale well as the number of > relations grow ( in cases of thousands or h

Re: another autovacuum scheduling thread

2025-10-23 Thread Sami Imseih
> On Fri, 24 Oct 2025 at 08:33, Sami Imseih wrote: > > Yeah, you’re correct, the list already exists; sorry I missed that. My > > main concern is > > the additional overhead of the sort operation, especially if we have > > many eligible > > tables and an aggressive autovacuum_naptime. > > It is tr

Re: another autovacuum scheduling thread

2025-10-23 Thread David Rowley
On Fri, 24 Oct 2025 at 08:33, Sami Imseih wrote: > Yeah, you’re correct, the list already exists; sorry I missed that. My > main concern is > the additional overhead of the sort operation, especially if we have > many eligible > tables and an aggressive autovacuum_naptime. It is true that there a

Re: another autovacuum scheduling thread

2025-10-23 Thread Sami Imseih
> On Thu, Oct 23, 2025 at 01:22:24PM -0500, Sami Imseih wrote: > > I was looking at v3, and I understand the formula will be updated in the > > next version. However, do you think we should benchmark the approach > > of using an intermediary list to store the eligible tables and sorting > > that li

Re: another autovacuum scheduling thread

2025-10-23 Thread Nathan Bossart
On Thu, Oct 23, 2025 at 01:22:24PM -0500, Sami Imseih wrote: > I was looking at v3, and I understand the formula will be updated in the > next version. However, do you think we should benchmark the approach > of using an intermediary list to store the eligible tables and sorting > that list, > whic

Re: another autovacuum scheduling thread

2025-10-23 Thread Sami Imseih
> > I think the one I proposed in [1] does this quite well. The table > > remains eligible to be autovacuumed with any score >= 1.0, and there's > > still a huge window of time to freeze a table once it's over > > autovacuum_freeze_max_age before there are issues and the exponential > > scaling onc

Re: another autovacuum scheduling thread

2025-10-22 Thread Nathan Bossart
On Thu, Oct 23, 2025 at 08:34:49AM +1300, David Rowley wrote: > On Thu, 23 Oct 2025 at 07:58, Nathan Bossart wrote: >> I'm imagining something a bit like the following: >> >> select xidage "age(relfrozenxid)", >> power(1.001, xidage::float8 / (select min_val >> from pg_settings where n

Re: another autovacuum scheduling thread

2025-10-22 Thread David Rowley
On Thu, 23 Oct 2025 at 07:58, Nathan Bossart wrote: > > That's a good point. I wonder if we should try to make the wraparound > > score independent of the *_freeze_max_age parameters (once the table age > > surpasses said parameters). Else, different settings will greatly impact > > how aggressi

Re: another autovacuum scheduling thread

2025-10-22 Thread Nathan Bossart
On Wed, Oct 22, 2025 at 01:40:11PM -0500, Nathan Bossart wrote: > On Wed, Oct 22, 2025 at 09:07:33AM +1300, David Rowley wrote: >> However, just thinking of non-standard setting... I do wonder if it'll >> be aggressive enough if someone did something like raise the >> *freeze_max_age to 1 billion (

Re: another autovacuum scheduling thread

2025-10-22 Thread Nathan Bossart
On Wed, Oct 22, 2025 at 09:07:33AM +1300, David Rowley wrote: > However, just thinking of non-standard setting... I do wonder if it'll > be aggressive enough if someone did something like raise the > *freeze_max_age to 1 billion (it's certainly common that people raise > this). With a 1.6 billion v

Re: another autovacuum scheduling thread

2025-10-21 Thread David Rowley
On Wed, 22 Oct 2025 at 03:38, Nathan Bossart wrote: > I did something similar to this in v3, although I used the *_freeze_max_age > parameters as the point to start scaling aggressively, and I simply raised > the score to the power of 10. > > I've yet to do any real testing with this stuff. I've

Re: another autovacuum scheduling thread

2025-10-21 Thread Nathan Bossart
On Sun, Oct 12, 2025 at 07:27:10PM +1300, David Rowley wrote: > On Sat, 11 Oct 2025 at 07:43, Robert Haas wrote: >> I think this is a reasonable starting point, although I'm surprised >> that you chose to combine the sub-scores using + rather than Max. > > Adding up the component scores doesn't m

Re: another autovacuum scheduling thread

2025-10-18 Thread Jeremy Schneider
On Wed, 8 Oct 2025 12:06:29 -0500 Sami Imseih wrote: > > One risk I see with this approach is that we will end up autovacuuming > tables that also take the longest time to complete, which could cause > smaller, quick-to-process tables to be neglected. > > It’s not always the case that the oldest

Re: another autovacuum scheduling thread

2025-10-18 Thread Jeremy Schneider
On Fri, 10 Oct 2025 16:24:51 -0400 Robert Haas wrote: > I don't think we > need something dramatically awesome to make a change to the status > quo, but if it's extremely easy to think up simple scenarios in which > a given idea will fail spectacularly, I'd be inclined to suspect that > there wil

Re: another autovacuum scheduling thread

2025-10-18 Thread Andres Freund
Hi, On 2025-10-08 10:18:17 -0500, Nathan Bossart wrote: > However, we do no such prioritization of the tables within a database. In > fact, the ordering of the tables is effectively random. We don't prioritize tables, but I don't think the order really is random? Isn't it basically in the order

Re: another autovacuum scheduling thread

2025-10-18 Thread David Rowley
On Thu, 9 Oct 2025 at 13:27, Jeremy Schneider wrote: > I'm arguing that it works well with autovacuum. Not saying there aren't > going to be certain workloads that it's suboptimal for. We're talking > about sorting by (M)XID age. As the clock continues to move forward any > table that doesn't get

Re: another autovacuum scheduling thread

2025-10-18 Thread Robert Haas
On Fri, Oct 10, 2025 at 1:31 PM Nathan Bossart wrote: > Here's a prototype of a "score" approach. Two notes: > > * I've given special priority to anti-wraparound vacuums. I think this is > important to avoid focusing too much on bloat when wraparound is imminent. > In any case, we need a separat

Re: another autovacuum scheduling thread

2025-10-18 Thread Jeremy Schneider
On Thu, 9 Oct 2025 12:59:23 +1300 David Rowley wrote: > I believe that is methodology for processing work applies much better > in scenarios where there's no new work continually arriving and > there's no adverse effects from giving a lower priority to certain > portions of the work. I don't thin

Re: another autovacuum scheduling thread

2025-10-18 Thread Nathan Bossart
On Wed, Oct 08, 2025 at 01:37:22PM -0400, Andres Freund wrote: > On 2025-10-08 10:18:17 -0500, Nathan Bossart wrote: >> The attached patch works by storing the maximum of the XID age and the MXID >> age in the list with the OIDs and sorting it prior to processing. > > I think it may be worth tryin

Re: another autovacuum scheduling thread

2025-10-18 Thread Jeremy Schneider
On Thu, 9 Oct 2025 14:03:34 +1300 David Rowley wrote: > I thought if we're to have a priority queue that it would be hard to > argue against sorting by how far over the given auto-vacuum threshold > that the table is. If you assume that a table that just meets the > dead rows required to trigger

another autovacuum scheduling thread

2025-10-18 Thread Nathan Bossart
/me dons flame-proof suit My goal with this thread is to produce some incremental autovacuum scheduling improvements for v19, but realistically speaking, I know that it's a bit of a long-shot. There have been many discussions over the years, and I've read through a few of them [0] [1] [2] [3] [4]

Re: another autovacuum scheduling thread

2025-10-18 Thread Nathan Bossart
On Thu, Oct 09, 2025 at 12:15:31PM -0400, Andres Freund wrote: > On 2025-10-09 11:01:16 -0500, Nathan Bossart wrote: >> I also wonder how hard it would be to gracefully catch the error and let >> the worker continue with the rest of its list... > > The main set of cases I've seen are when workers

Re: another autovacuum scheduling thread

2025-10-18 Thread David Rowley
On Thu, 9 Oct 2025 at 12:41, Jeremy Schneider wrote: > I think an approach of doing largest objects first actually might work > really well for balancing work amongst autovacuum workers. Many years > ago I designed a system to backup many databases with a pool of workers > and used this same simpl

Re: another autovacuum scheduling thread

2025-10-17 Thread Nathan Bossart
Thanks for taking a look. On Fri, Oct 10, 2025 at 02:42:57PM -0400, Robert Haas wrote: > I think this is a reasonable starting point, although I'm surprised > that you chose to combine the sub-scores using + rather than Max. My thinking was that we should consider as many factors as we can in the

Re: another autovacuum scheduling thread

2025-10-17 Thread David Rowley
On Thu, 9 Oct 2025 at 14:47, Jeremy Schneider wrote: > > On Wed, 8 Oct 2025 18:25:20 -0700 > Jeremy Schneider wrote: > > If users are tuning this thing then I feel like we've already lost the > > battle :) > > I replied too quickly. Re-reading your email, I think your proposing a > different algo

Re: another autovacuum scheduling thread

2025-10-17 Thread Robert Haas
On Fri, Oct 10, 2025 at 3:44 PM Nathan Bossart wrote: > On Fri, Oct 10, 2025 at 02:42:57PM -0400, Robert Haas wrote: > > I think this is a reasonable starting point, although I'm surprised > > that you chose to combine the sub-scores using + rather than Max. > > My thinking was that we should cons

Re: another autovacuum scheduling thread

2025-10-17 Thread Jeremy Schneider
On Wed, 8 Oct 2025 18:25:20 -0700 Jeremy Schneider wrote: > On Thu, 9 Oct 2025 14:03:34 +1300 > David Rowley wrote: > > > I thought if we're to have a priority queue that it would be hard to > > argue against sorting by how far over the given auto-vacuum > > threshold that the table is. If you

Re: another autovacuum scheduling thread

2025-10-17 Thread Andres Freund
Hi, On 2025-10-09 11:01:16 -0500, Nathan Bossart wrote: > On Wed, Oct 08, 2025 at 01:37:22PM -0400, Andres Freund wrote: > > On 2025-10-08 10:18:17 -0500, Nathan Bossart wrote: > >> The attached patch works by storing the maximum of the XID age and the MXID > >> age in the list with the OIDs and s

Re: another autovacuum scheduling thread

2025-10-17 Thread David Rowley
On Sat, 11 Oct 2025 at 07:43, Robert Haas wrote: > I think this is a reasonable starting point, although I'm surprised > that you chose to combine the sub-scores using + rather than Max. Adding up the component scores doesn't make sense to me either. That means you could have 0.5 for inserted tup

Re: another autovacuum scheduling thread

2025-10-13 Thread Nathan Bossart
On Thu, Oct 09, 2025 at 11:13:48AM -0500, Nathan Bossart wrote: > On Thu, Oct 09, 2025 at 04:13:23PM +1300, David Rowley wrote: >> I think the best way to understand it is if you look at >> relation_needs_vacanalyze() and see how it calculates boolean values >> for boolean output params. So, instea

Re: another autovacuum scheduling thread

2025-10-13 Thread Robert Haas
On Fri, Oct 10, 2025 at 6:00 PM Jeremy Schneider wrote: > The spectacular failures I've seen with autovac usually come down to > things like too much sleeping (cost_delay) or too few workers, where > better ordering would be nice but probably wouldn't fix any real > problems leading to the spectac

Re: another autovacuum scheduling thread

2025-10-09 Thread Peter Geoghegan
On Thu, Oct 9, 2025 at 12:15 PM Andres Freund wrote: > > Each worker would consult this table before processing. If the table is > > there, it would remove it from the shared table and skip processing it. > > Then the next worker would try processing the table again. > > > > I also wonder how har

Re: another autovacuum scheduling thread

2025-10-09 Thread Nathan Bossart
On Thu, Oct 09, 2025 at 04:13:23PM +1300, David Rowley wrote: > I think the best way to understand it is if you look at > relation_needs_vacanalyze() and see how it calculates boolean values > for boolean output params. So, instead of calculating just a boolean > value it instead calculates a float

Re: another autovacuum scheduling thread

2025-10-08 Thread Jeremy Schneider
On Wed, 8 Oct 2025 17:27:27 -0700 Jeremy Schneider wrote: > On Thu, 9 Oct 2025 12:59:23 +1300 > David Rowley wrote: > > > I believe that is methodology for processing work applies much > > better in scenarios where there's no new work continually arriving > > and there's no adverse effects from

Re: another autovacuum scheduling thread

2025-10-08 Thread Sami Imseih
> Not saying that the current approach, which is as you mention is > random, is any better, however this approach will likely increase > the behavior of large tables saturating workers. Maybe it will be good to allocate some workers to the oldest tables and workers based on some random list? This

Re: another autovacuum scheduling thread

2025-10-08 Thread Álvaro Herrera
On 2025-Oct-08, Sami Imseih wrote: > One risk I see with this approach is that we will end up autovacuuming > tables that also take the longest time to complete, which could cause > smaller, quick-to-process tables to be neglected. Perhaps we can have autovacuum workers decide on a mode to use at

Re: another autovacuum scheduling thread

2025-10-08 Thread Sami Imseih
Thanks for raising this topic! I agree that autovacuum scheduling could be improved. > * Prioritizing tables based on their (M)XID age might help avoid more > aggressive vacuums, not to mention wraparound. Of course, there are > scenarios where this doesn't work. For example, the age of a table