Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-02-04 Thread Peter Geoghegan
dge of running out of memory. I think in general > people tend to avoid such things in benchmarking scenarios, but even > if include stuff like this, it's hard to know what to include that > would be representative of real life, because just about anything > *could* happen in real life. Then what could you have confidence in? -- Peter Geoghegan

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-02-04 Thread Peter Geoghegan
eal to the actual behavior of the system. In particular, there is really only one way that the free space management can work for the two big tables that will perform acceptably -- the orders have to be stored in the same place to begin with, and stay in the same place forever (at least to the extent that that's possible). -- Peter Geoghegan

Re: should vacuum's first heap pass be read-only?

2022-02-04 Thread Peter Geoghegan
r of dead items that we now need to remove from indexes, as well as metadata from any preexisting conveyor belt. -- Peter Geoghegan

Re: decoupling table and index vacuum

2022-02-04 Thread Peter Geoghegan
or, since _bt_check_unique() tends to do a good job of setting LP_DEAD bits, independent of the kill_prior_tuple thing. You can avoid using kill_prior_tuple by forcing bitmap scans, of course. -- Peter Geoghegan

Re: Stats collector's idx_blks_hit value is highly misleading in practice

2022-02-04 Thread Peter Geoghegan
On Thu, Feb 3, 2022 at 7:08 PM John Naylor wrote: > Is this a TODO candidate? What would be a succinct title for it? I definitely think that it's worth working on. I suppose it follows that it should go on the TODO list. -- Peter Geoghegan

Re: Why is INSERT-driven autovacuuming based on pg_class.reltuples?

2022-02-02 Thread Peter Geoghegan
on in the table). So that also recommends "relativistically interpreting" the values later on. This specific issue has even less to do with autovacuum_vacuum_scale_factor than the main point, of course. I agree with you that the insert-driven stuff isn't a special case. -- Peter Geoghegan

Re: Why is INSERT-driven autovacuuming based on pg_class.reltuples?

2022-01-28 Thread Peter Geoghegan
ted to this messiness with statistics and pg_class. -- Peter Geoghegan

Re: Why is INSERT-driven autovacuuming based on pg_class.reltuples?

2022-01-27 Thread Peter Geoghegan
On Thu, Jan 27, 2022 at 12:20 PM Peter Geoghegan wrote: > Both VACUUM and ANALYZE update pg_class.reltuples. But this code seems > to assume that it's only something that VACUUM can ever do. Why > wouldn't we expect a plain ANALYZE to have actually been the last > thi

Why is INSERT-driven autovacuuming based on pg_class.reltuples?

2022-01-27 Thread Peter Geoghegan
append-only table, relative to the documented behavior of autovacuum_vacuum_insert_scale_factor? -- Peter Geoghegan

Re: autovacuum prioritization

2022-01-26 Thread Peter Geoghegan
inherent futility of an anti-wraparound VACUUM that runs against a table whose relfrozenxid is already exactly equal to the VACUUM's OldestXmin (say because of a leaked replication slot -- anything that makes vacuuming fundamentally unable to advance relfrozenxid, really). -- Peter Geoghegan

Re: autovacuum prioritization

2022-01-26 Thread Peter Geoghegan
> On Wed, Jan 26, 2022 at 10:55 AM Robert Haas wrote: > On Tue, Jan 25, 2022 at 3:32 PM Peter Geoghegan wrote: > > For example, a > > page that has 5 dead heap-only tuples is vastly different to a similar > > page that has 5 LP_DEAD items instead -- and yet our cu

Re: autovacuum prioritization

2022-01-25 Thread Peter Geoghegan
uming one large table at a time. Also might make sense to strategically align the work with the beginning of a new checkpoint. -- Peter Geoghegan

Re: autovacuum prioritization

2022-01-22 Thread Peter Geoghegan
out percentage table growth). Constant mini-vacuums against such a table make sense, since costs are almost exactly proportional to the number of heap pages appended since the last VACUUM. -- Peter Geoghegan

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-01-21 Thread Peter Geoghegan
is either empty or has only frozen tuples, VACUUM will set relfrozenxid to oldestxmin/removable cutoff. Internally, oldestxmin is the "starting point" for our final/target relfrozenxid for the table. We ratchet it back dynamically, whenever we see an older-than-current-target XID that cannot be immediately frozen (e.g., when we can't easily get a cleanup lock on the page). -- Peter Geoghegan

Re: autovacuum prioritization

2022-01-20 Thread Peter Geoghegan
ffective techniques is to manually VACUUM when the system is naturally idle, like at night time. If that could be quasi-automated, or if the criteria used by autovacuum scheduling gave just a little weight to how busy the system is right now, then we would have more slack when the system becomes very busy. -- Peter Geoghegan

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-01-20 Thread Peter Geoghegan
fault should be changed to -1, which is interpreted as "whatever autovacuum_freeze_max_age/2 is". But it should still be greatly deemphasized in user docs. -- Peter Geoghegan

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-01-19 Thread Peter Geoghegan
y we derive FreezeLimit from autovacuum_freeze_max_age/2 (instead of vacuum_freeze_min_age). We still ought to have the opportunity to wait for the cleanup lock for rather a long time -- if the XID consumption rate is so high that that isn't true, then we're doomed anyway. All told, there seems to be a huge net reduction in risk with this design. -- Peter Geoghegan

Re: A qsort template

2022-01-18 Thread Peter Geoghegan
s with abbreviated keys encoded as unsigned integers. Just a thought. -- Peter Geoghegan

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-01-18 Thread Peter Geoghegan
On Tue, Jan 18, 2022 at 6:11 AM Robert Haas wrote: > On Tue, Jan 18, 2022 at 12:14 AM Peter Geoghegan wrote: > > I quite clearly said that you'll only get an anti-wraparound VACUUM > > with the patch applied when the only factor that *ever* causes *any* > > autovacuum worke

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-01-17 Thread Peter Geoghegan
On Mon, Jan 17, 2022 at 8:13 PM Robert Haas wrote: > On Mon, Jan 17, 2022 at 5:41 PM Peter Geoghegan wrote: > > That just seems like semantics to me. The very next sentence after the > > one you quoted in your reply was "And so it's highly unlikely that any > > given VA

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-01-17 Thread Peter Geoghegan
On Mon, Jan 17, 2022 at 2:13 PM Robert Haas wrote: > On Mon, Jan 17, 2022 at 4:28 PM Peter Geoghegan wrote: > > Updating relfrozenxid should now be thought of as a continuous thing, > > not a discrete thing. > > I think that's pretty nearly 100% wrong. The most simplisti

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-01-17 Thread Peter Geoghegan
On Mon, Jan 17, 2022 at 7:12 AM Robert Haas wrote: > On Thu, Jan 13, 2022 at 4:27 PM Peter Geoghegan wrote: > > 1. Cases where our inability to get a cleanup lock signifies nothing > > at all about the page in question, or any page in the same table, with > > the sam

Time to increase hash_mem_multiplier default?

2022-01-16 Thread Peter Geoghegan
of. This effect is reliable (data cardinality won't matter, for example). And so the improvement that is possible from giving a sort more memory is far smaller than (say) the improvement in performance we typically see when the optimizer switches from a hash aggregate to a group aggregate. -- Peter Geoghegan

Re: Unifying VACUUM VERBOSE and log_autovacuum_min_duration output

2022-01-14 Thread Peter Geoghegan
On Thu, Jan 6, 2022 at 10:21 AM Peter Geoghegan wrote: > The patch bitrot again, so attached is a rebased version, v3. I pushed a version of this patch earlier. This final version didn't go quite as far as v3 did: it retained a few VACUUM VERBOSE only INFO messages (it didn't demote t

Re: small development tip: Consider using the gold linker

2022-01-13 Thread Peter Geoghegan
On Fri, Jul 20, 2018 at 8:16 PM Peter Geoghegan wrote: > On Mon, Jul 9, 2018 at 4:40 PM, Andres Freund wrote: > > FWIW, I've lately noticed that I spend a fair time waiting for the > > linker during edit-compile-test cycles. Due to an independent issue I > > just

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-01-13 Thread Peter Geoghegan
t autovacuum can no longer advance relfrozenxid (using the enhanced log output) where that's generally expected. -- Peter Geoghegan

Re: UNIQUE null treatment option

2022-01-13 Thread Peter Geoghegan
On Thu, Jan 13, 2022 at 10:36 AM Peter Geoghegan wrote: > I wonder if the logic for setting BTScanInsertData.anynullkeys inside > _bt_mkscankey() is the place to put your test for > rel->rd_index->indnullsnotdistinct -- not inside _bt_doinsert(). That > would probably ne

Re: UNIQUE null treatment option

2022-01-13 Thread Peter Geoghegan
ay. This feels more natural to me because a NULL key column in a NULLS NOT DISTINCT unique constraint is very similar to a NULL non-key column in an INCLUDE index, as far as our requirements go -- and so both cases should probably be dealt with at the same point. -- Peter Geoghegan

Re: disfavoring unparameterized nested loops

2022-01-12 Thread Peter Geoghegan
get back to it, FWIW. -- Peter Geoghegan

Re: do only critical work during single-user vacuum?

2022-01-11 Thread Peter Geoghegan
Id() WARNINGs ought to be changed to reference VACUUM LIMIT. (You probably just didn't get around to that in this POC, but couldn't hurt to remind you.) -- Peter Geoghegan

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-01-07 Thread Peter Geoghegan
o hit > the backstop all the time. We want it to be made of, you know, nylon > twine, rather than, say, sharp nails. :-) Absolutely. But if autovacuum can only ever run due to age(relfrozenxid) reaching autovacuum_freeze_max_age, then I can't see a downside. Again, the v5-0005-*patch needs to meet the standard that I've laid out. If it doesn't then I've messed up already. -- Peter Geoghegan

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-01-06 Thread Peter Geoghegan
On Thu, Jan 6, 2022 at 2:45 PM Peter Geoghegan wrote: > But the "freeze early" heuristics work a bit like that anyway. We > won't freeze all the tuples on a whole heap page early if we won't > otherwise set the heap page to all-visible (not all-frozen) in the VM &

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-01-06 Thread Peter Geoghegan
On Thu, Jan 6, 2022 at 12:54 PM Robert Haas wrote: > On Fri, Dec 17, 2021 at 9:30 PM Peter Geoghegan wrote: > > Can we fully get rid of vacuum_freeze_table_age? Maybe even get rid of > > vacuum_freeze_min_age, too? Freezing tuples is a maintenance task for > > physical block

Re: Add 64-bit XIDs into PostgreSQL 15

2022-01-06 Thread Peter Geoghegan
and the FSM) to make all this work. [1] https://www.postgresql.org/message-id/CAH2-Wz%3DzEV4y_wxh-A_EvKxeAoCMdquYMHABEh_kZO1rk3a-gw%40mail.gmail.com -- Peter Geoghegan

Re: Unifying VACUUM VERBOSE and log_autovacuum_min_duration output

2022-01-06 Thread Peter Geoghegan
error_info() call (or any other call that doesn't set a saved_err_info). -- Peter Geoghegan v3-0001-Unify-VACUUM-VERBOSE-and-log_autovacuum_min_durat.patch Description: Binary data

Re: Unifying VACUUM VERBOSE and log_autovacuum_min_duration output

2021-12-22 Thread Peter Geoghegan
ntation (a problem that I'm trying to permanently avoid by unifying everything). While things have improved a lot here recently, I don't think that things have fully settled yet -- the output will probably change quite a bit more in Postgres 15. That makes me a little hesitant to promise very

Re: do only critical work during single-user vacuum?

2021-12-22 Thread Peter Geoghegan
ing that makes generic choices, that are not way too conservative, but also don't risk making the problem worse instead of better. -- Peter Geoghegan

Re: track_io_timing default setting

2021-12-22 Thread Peter Geoghegan
getting the time, not measuring elapsed time. It seems reasonable to suppose that an operating system that offers a facility for measuring elapsed time won't have horrible performance problems. clock_gettime() first appeared in POSIX almost 30 years ago. -- Peter Geoghegan

Re: do only critical work during single-user vacuum?

2021-12-21 Thread Peter Geoghegan
esn't accept any tables). The only reason to have this is for scenarios where the user is already in a tough spot with wraparound failure, like that client of yours. Nobody wants to force the failsafe for one specific table. It's not general purpose, at all, and shouldn't claim to be. -- Peter Geoghegan

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2021-12-20 Thread Peter Geoghegan
s that we still expect an aggressive VACUUM (with the patch) are the cases where there is zero chance of that happening. Almost by definition. > Given the opportunistic freezing, that's true but I'm concerned > whether opportunistic freezing always works well on all tables since > freezing tuples is not 0 cost. That is the big question for this patch. -- Peter Geoghegan

Re: do only critical work during single-user vacuum?

2021-12-20 Thread Peter Geoghegan
t just the table (or tables) whose relfrozenxid is very old. That's still naive, but much less naive than simply telling users to VACUUM the whole database in single user mode while vacuuming indexes, etc. -- Peter Geoghegan

Re: do only critical work during single-user vacuum?

2021-12-20 Thread Peter Geoghegan
ched as part of a more worked out solution (assuming it didn't fully remove single user mode from the equation, which would be better still). -- Peter Geoghegan v1-0001-Always-trigger-failsafe-in-single-user-mode.patch Description: Binary data

Re: Add index scan progress to pg_stat_progress_vacuum

2021-12-20 Thread Peter Geoghegan
e have to do something called backtracking in btvacuumpage(), which you've ignored -- that's another reasonably common way that we'll end up scanning a page twice. But that probably should just be ignored -- it's too narrow a case to be worth caring about. -- Peter Geoghegan

Re: Add index scan progress to pg_stat_progress_vacuum

2021-12-20 Thread Peter Geoghegan
led options #2 and #3, but those options still seem unappealing to me. -- Peter Geoghegan

Re: Unifying VACUUM VERBOSE and log_autovacuum_min_duration output

2021-12-20 Thread Peter Geoghegan
On Mon, Nov 29, 2021 at 6:51 PM Peter Geoghegan wrote: > Attached is a WIP patch doing this. This has bitrot, so I attach v2, mostly just to keep the CFTester status green. The only real change is one minor simplification to how we set everything up, inside heap_vacuum_rel(). -- Pe

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2021-12-17 Thread Peter Geoghegan
136cf6e3, shortly after the original visibility map implementation was committed in 2009. The idea that it helps us to advance relfrozenxid outside of aggressive VACUUMs (per commit message from bf136cf6e3) seems like it might no longer matter with the patch -- because now we won't ever set a page all-visible but not all-frozen. Plus the idea that we need to do all this work just to get readahead from the OS seems...questionable. -- Peter Geoghegan

Re: parallel vacuum comments

2021-12-14 Thread Peter Geoghegan
ration to heap vacuuming. With a distinct autovacuum schedule (well, the schedule would be related, but still distinct). -- Peter Geoghegan

Re: generalized conveyor belt storage

2021-12-14 Thread Peter Geoghegan
ting the TIDs into a store used for dead TIDs could make sense. Especially in extreme cases. -- Peter Geoghegan

Re: Defining (and possibly skipping) useless VACUUM operations

2021-12-14 Thread Peter Geoghegan
over time -- I'm not sure that 20% is much worse than 60%, or much better than 5% (very hard to model it). But if we make the high-level, abstract idea of "aggressiveness" more of a continuous thing, and not something that's defined by sharp (and largely meaningless) XID-based cutoffs, we have every chance of nipping these problems in the bud (without needing to model much of anything). -- Peter Geoghegan

Re: Defining (and possibly skipping) useless VACUUM operations

2021-12-14 Thread Peter Geoghegan
s concerned with bloat. I think that you're right about that: the question there must be "why are we even launching these bloat-orientated autovacuums that actually find no bloat?". -- Peter Geoghegan

Defining (and possibly skipping) useless VACUUM operations

2021-12-12 Thread Peter Geoghegan
CUUM operation" in a way that is both reliable and proportionate. So just logging something necessitates solving that hard problem. [1] https://commitfest.postgresql.org/36/3433/ -- Peter Geoghegan

Re: Unifying VACUUM VERBOSE and log_autovacuum_min_duration output

2021-12-11 Thread Peter Geoghegan
g processed. Various statistics about the tables are printed as well." Having 2 ereports (not just 1) isn't essential, but it seems useful because it makes the new VACUUM VERBOSE continue to work like this. But without any of the downsides that go with seeing way too much detail, moment to moment. -- Peter Geoghegan

Re: Unifying VACUUM VERBOSE and log_autovacuum_min_duration output

2021-12-11 Thread Peter Geoghegan
On Sat, Dec 11, 2021 at 1:13 PM Peter Geoghegan wrote: > Yes -- in order to report aggressiveness in VACUUM VERBOSE. But the > autovacuum case still reports verbose-ness, in the same way as it > always has -- in that same LOG entry. We don't want to repeat > ourselves in the VERBOSE

Re: Unifying VACUUM VERBOSE and log_autovacuum_min_duration output

2021-12-11 Thread Peter Geoghegan
ant to repeat ourselves in the VERBOSE case, which will have already indicated its verboseness in the up-front ereport(). In other words, every distinct case reports on its aggressiveness exactly once per call into lazyvacuum.c. In roughly the same way as it works on HEAD. -- Peter Geoghegan

Re: Unifying VACUUM VERBOSE and log_autovacuum_min_duration output

2021-12-11 Thread Peter Geoghegan
moted to DEBUG2. What do you think of that aspect? I could add some the getrusage output back where that makes sense. I don't have very strong feelings about that. -- Peter Geoghegan

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2021-12-10 Thread Peter Geoghegan
On Tue, Nov 30, 2021 at 11:52 AM Peter Geoghegan wrote: > I haven't had time to work through any of your feedback just yet -- > though it's certainly a priority for. I won't get to it until I return > home from PGConf NYC next week. Attached is v3, which works through most of you

Re: do only critical work during single-user vacuum?

2021-12-09 Thread Peter Geoghegan
able to allocate XIDs again -- but not too much more (not until things are back to normal). Now the plan is starting to get ambitious relative to how often it'll be seen by users, though. -- Peter Geoghegan

Re: do only critical work during single-user vacuum?

2021-12-09 Thread Peter Geoghegan
till have a separate option for the postgres executable, just to invoke single-user mode and call this function. It would mostly just be window dressing, of course, but that still seems useful. -- Peter Geoghegan

Re: do only critical work during single-user vacuum?

2021-12-09 Thread Peter Geoghegan
vacuum_multixact_freeze_max_age) settings, with sanity checks. We could then apply this criteria in new code that implements this "big red button" (maybe this is a new option for the postgres executable, a little like --single?). Something that's reasonably targeted, and dead simple to use. -- Peter Geoghegan

Re: do only critical work during single-user vacuum?

2021-12-09 Thread Peter Geoghegan
On Thu, Dec 9, 2021 at 1:04 PM Peter Geoghegan wrote: > On Thu, Dec 9, 2021 at 11:28 AM John Naylor > wrote: > > Now that we have a concept of a fail-safe vacuum, maybe it would be > > beneficial to skip a vacuum in single-user mode if the fail-safe > > criteria were

Re: do only critical work during single-user vacuum?

2021-12-09 Thread Peter Geoghegan
here is an argument for it. -- Peter Geoghegan

Re: Why doesn't GiST VACUUM require a super-exclusive lock, like nbtree VACUUM?

2021-12-08 Thread Peter Geoghegan
On Tue, Nov 30, 2021 at 5:09 PM Peter Geoghegan wrote: > Attached draft patch attempts to explain things in this area within > the nbtree README. There is a much shorter comment about it within > vacuumlazy.c. I am concerned about GiST index-only scans themselves, > of course, but

Re: Why doesn't pgstat_report_analyze() focus on not-all-visible-page dead tuple counts, specifically?

2021-12-07 Thread Peter Geoghegan
"bloat units" a probability density function, and not just a simple expected value. Teaching the autovacuum.c scheduler to distinguish signal from noise could be very valuable, if it enables opportunistic batching of work, or off-hours work. We don't have to respect noise. The devil is in the details, of course. -- Peter Geoghegan

Re: Why doesn't pgstat_report_analyze() focus on not-all-visible-page dead tuple counts, specifically?

2021-12-07 Thread Peter Geoghegan
n abstract units -- we need to call those abstract units *something*. I don't care what that name ends up being, as long as it reflects reality. -- Peter Geoghegan

Re: Why doesn't pgstat_report_analyze() focus on not-all-visible-page dead tuple counts, specifically?

2021-12-07 Thread Peter Geoghegan
urate, but it's also important to be tolerant of model error, which is inevitable. We should make pragmatic decisions about what kinds of errors our new model will have. And it should have at least a rudimentary ability to learn from its mistakes. -- Peter Geoghegan

Re: Why doesn't pgstat_report_analyze() focus on not-all-visible-page dead tuple counts, specifically?

2021-12-06 Thread Peter Geoghegan
a medium term solution. Something that considers the concentration of dead tuples in heap pages seems promising. -- Peter Geoghegan

Re: Why doesn't pgstat_report_analyze() focus on not-all-visible-page dead tuple counts, specifically?

2021-12-06 Thread Peter Geoghegan
On Mon, Dec 6, 2021 at 2:37 PM Peter Geoghegan wrote: > On Mon, Dec 6, 2021 at 12:07 PM Robert Haas wrote: > > So does this. If some of the table is now all-visible when it wasn't > > before, it's certainly a good guess that the portions that still > > aren't have about

Re: Why doesn't pgstat_report_analyze() focus on not-all-visible-page dead tuple counts, specifically?

2021-12-06 Thread Peter Geoghegan
here are very few or even 0 all-visible pages), it wouldn't hurt us at all, and we wouldn't need to have special pgstat_report_vacuum()-only heuristics. -- Peter Geoghegan

Why doesn't pgstat_report_analyze() focus on not-all-visible-page dead tuple counts, specifically?

2021-12-04 Thread Peter Geoghegan
eIsAllVisible() pages? I think that it shouldn't be too hard to come up with solutions to those problems (the extrapolation idea already hints at a solution), but for now I'd like to keep the discussion high level. [1] https://postgr.es/m/CAH2-Wz=9r83wcwzcpuh4fvpedm4znzbzmvp3rt21+xhqwmu...@mail.gmail.com -- Peter Geoghegan

Re: Skip vacuum log report code in lazy_scan_heap() if possible

2021-12-03 Thread Peter Geoghegan
If my patch to unite vacuum verbose and the autovacuum logging gets in, then this issue also goes away. Peter Geoghegan (Sent from my phone)

Re: Why doesn't GiST VACUUM require a super-exclusive lock, like nbtree VACUUM?

2021-11-30 Thread Peter Geoghegan
On Tue, Nov 30, 2021 at 5:09 PM Peter Geoghegan wrote: > I believe that there have been several historic reasons why we need a > cleanup lock during nbtree VACUUM, and that there is only one > remaining reason for it today. So the history is unusually complicated. Minor correction: we

Re: Why doesn't GiST VACUUM require a super-exclusive lock, like nbtree VACUUM?

2021-11-30 Thread Peter Geoghegan
On Fri, Nov 5, 2021 at 3:26 AM Andrey Borodin wrote: > > 4 нояб. 2021 г., в 20:58, Peter Geoghegan написал(а): > > That's a pretty unlikely scenario. And even > > if it happened it would only happen once (until the next time we get > > unlucky). What are the chances of

Deprecating the term "super-exclusive lock"

2021-11-30 Thread Peter Geoghegan
own, but in a world with something like Robert Haas's conveyor belt design (a world with *selective* index vacuuming), it could be quite valuable. [1] https://postgr.es/m/CAH2-Wz=PqOziyRSrnN5jAtfXWXY7-BJcHz9S355LH8Dt=5q...@mail.gmail.com -- Peter Geoghegan v1-0001-Deprecate-the-term-super-exclusive-lock.patch Description: Binary data

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2021-11-30 Thread Peter Geoghegan
On Tue, Nov 23, 2021 at 5:01 PM Peter Geoghegan wrote: > > Behaviour that lead to a "sudden" falling over, rather than getting > > gradually > > worse are bad - they somehow tend to happen on Friday evenings :). > > These are among our most important challenge

Re: Rename dead_tuples to dead_items in vacuumlazy.c

2021-11-30 Thread Peter Geoghegan
ssumptions are met (even these assumptions only really work with a properly normalized database schema). Maybe revising the model used for autovacuum scheduling wouldn't include changing pg_stat_progress_vacuum, since that isn't technically "part of the model" --- I'm not sure. But it's not something that I am in a hurry to fix. -- Peter Geoghegan

Re: Unifying VACUUM VERBOSE and log_autovacuum_min_duration output

2021-11-29 Thread Peter Geoghegan
ch harder, and not likely to happen soon. [1] https://postgr.es/m/CAH2-Wznp=c=Opj8Z7RMR3G=ec3_jfgymn_yvmcejopchzwb...@mail.gmail.com -- Peter Geoghegan

Re: Unifying VACUUM VERBOSE and log_autovacuum_min_duration output

2021-11-29 Thread Peter Geoghegan
On Fri, Nov 26, 2021 at 12:37 PM Peter Geoghegan wrote: > My preferred approach to this is simple: redefine VACUUM VERBOSE to > not show incremental output, which seems rather unhelpful anyway. This > does mean that VACUUM VERBOSE won't show certain information that > might occasional

Re: Rename dead_tuples to dead_items in vacuumlazy.c

2021-11-29 Thread Peter Geoghegan
h that still uses both terms when talking about dead_items. But the final commit actually makes it clear why, in comments above the LVDeadItems struct itself: LVDeadItems is used by both index vacuuming and heap vacuuming. Thanks -- Peter Geoghegan

Re: Unifying VACUUM VERBOSE and log_autovacuum_min_duration output

2021-11-26 Thread Peter Geoghegan
more technical debt in this area for Postgres 15. -- Peter Geoghegan

Re: Unifying VACUUM VERBOSE and log_autovacuum_min_duration output

2021-11-26 Thread Peter Geoghegan
On Fri, Nov 26, 2021 at 12:37 PM Peter Geoghegan wrote: > Unifying everything cannot be approached mechanically, so doing this > requires real buy-in. It's a bit tricky because VACUUM VERBOSE is > supposed to show real time information about what just finished, as a > kind of rudiment

Unifying VACUUM VERBOSE and log_autovacuum_min_duration output

2021-11-26 Thread Peter Geoghegan
ion output by VACUUM VERBOSE just isn't useful outside of debugging scenarios -- it just isn't actionable to users (users only really care about how much smaller the table became through truncation). The low level details could easily be output as DEBUG1 (not DEBUG2) instead. Thoughts? -- Peter Geoghegan

Re: Why not try for a HOT update, even when PageIsFull()?

2021-11-26 Thread Peter Geoghegan
On Sun, Nov 21, 2021 at 4:29 PM Peter Geoghegan wrote: > On Fri, Nov 19, 2021 at 11:51 AM Alvaro Herrera > wrote: > > I certainly do not object to removing it. > > I'd like to do so soon. I'll wait a few more days, in case Pavan objects. Pushed just now. Thanks -- Peter Geoghegan

Re: Rename dead_tuples to dead_items in vacuumlazy.c

2021-11-24 Thread Peter Geoghegan
u agree with me that dead_tuples is now a bad name). This one detail affects more individual lines of code than the restructuring of comments. -- Peter Geoghegan

Re: Rename dead_tuples to dead_items in vacuumlazy.c

2021-11-24 Thread Peter Geoghegan
ng special cases in vacuumlazy.c, ISTM that we ought to do everything except pruning itself (and freezing tuples, which effectively depends on pruning) without even acquiring a cleanup lock. Which is actually quite a lot of things. -- Peter Geoghegan

Rename dead_tuples to dead_items in vacuumlazy.c

2021-11-23 Thread Peter Geoghegan
, but wasn't yet broken up into smaller, better-scoped functions. If there are no objections, I'll move on this soon. It's mostly just mechanical changes. -- Peter Geoghegan v1-0001-vacuumlazy.c-Rename-dead_tuples-to-dead_items.patch Description: Binary data

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2021-11-23 Thread Peter Geoghegan
irtuous cycle that totally prevents us from getting an aggressive VACUUM even once. > > I have occasionally wondered if the whole idea of reading heap pages > > with only a pin (and having cleanup locks in VACUUM) is really worth > > it -- alternative designs seem possible. Obviously that's a BIG > > discussion, and not one to have right now. But it seems kind of > > relevant. > > With 'reading' do you mean reads-from-os, or just references to buffer > contents? The latter. [1] https://postgr.es/m/CAH2-Wz=9r83wcwzcpuh4fvpedm4znzbzmvp3rt21+xhqwmu...@mail.gmail.com -- Peter Geoghegan

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2021-11-22 Thread Peter Geoghegan
gt; > + * Now save details of the LP_DEAD items from the page in the > > dead_tuples > > + * array iff VACUUM uses two-pass strategy case > > + */ > > Do we really need to have separate code for this in lazy_scan_prune() and > lazy_scan_noprune()? There is hardly any repetition, though. > > + } > > + else > > + { > > + /* > > + * We opt to skip FSM processing for the page on the grounds > > that it > > + * is probably being modified by concurrent DML operations. > > Seems > > + * best to assume that the space is best left behind for > > future > > + * updates of existing tuples. This matches what > > opportunistic > > + * pruning does. > > Why can we assume that there concurrent DML rather than concurrent read-only > operations? IME it's much more common for read-only operations to block > cleanup locks than read-write ones (partially because the frequency makes it > easier, partially because cursors allow long-held pins, partially because the > EXCLUSIVE lock of a r/w operation wouldn't let us get here) I actually agree. It still probably isn't worth dealing with the FSM here, though. It's just too much mechanism for too little benefit in a very rare case. What do you think? -- Peter Geoghegan

Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2021-11-21 Thread Peter Geoghegan
the level of each heap page. Of course we *need* to prune sometimes (e.g., might be necessary to do so to set the page all-visible in the visibility map), but why bother when we don't, and when there is no reason to think that it'll help anyway? Something to think about, at least. -- Peter Geoghegan

Re: Why not try for a HOT update, even when PageIsFull()?

2021-11-21 Thread Peter Geoghegan
e to do so soon. I'll wait a few more days, in case Pavan objects. Thanks -- Peter Geoghegan

Why not try for a HOT update, even when PageIsFull()?

2021-11-17 Thread Peter Geoghegan
would have found it easier to debug an issue that led to a PANIC inside heap_update() earlier this year (see commit 34f581c39e). That bug was judged to be an old bug in heap_update(), but we only started to see PANICs when the aforementioned enhancement to VACUUM went in. -- Peter Geoghegan v1

Re: Is heap_page_prune() stats collector accounting wrong?

2021-11-12 Thread Peter Geoghegan
On Fri, Nov 12, 2021 at 11:29 AM Peter Geoghegan wrote: > We compensate here precisely because we are not running in VACUUM (it > has to be an opportunistic prune in practice). > If we're not running in VACUUM, and have to make a statistics > collector call, then we don't want to

Re: Is heap_page_prune() stats collector accounting wrong?

2021-11-12 Thread Peter Geoghegan
On Fri, Nov 12, 2021 at 10:45 AM Peter Geoghegan wrote: > Let's assume that somehow I have it wrong. Even then, why should we > compensate like this for the stats collector, but not for VACUUM? > There's certainly no corresponding code in vacuumlazy.c that does a > similar tr

Is heap_page_prune() stats collector accounting wrong?

2021-11-12 Thread Peter Geoghegan
ke this for the stats collector, but not for VACUUM? There's certainly no corresponding code in vacuumlazy.c that does a similar transformation with ndeleted. -- Peter Geoghegan

Re: 2021-11-11 release announcement draft

2021-11-10 Thread Peter Geoghegan
even heapallindexed only notices the absence of needed index tuples. Whereas the VACUUM bug is mostly (but not entirely) characterized by the *presence* of "extra", corrupt index tuples -- tuples that VACUUM should have already removed, but didn't. -- Peter Geoghegan

Re: amcheck's verify_heapam(), and HOT chain verification

2021-11-09 Thread Peter Geoghegan
thinking of the prepared xact CIC bugs found using amcheck). The invariants that heapallindexed tests generalize to many different situations, including situations that I couldn't have possibly anticipated with any kind of precision. Having the right high level idea is what really mattered. -- Peter Geoghegan

Re: amcheck's verify_heapam(), and HOT chain verification

2021-11-06 Thread Peter Geoghegan
On Fri, Nov 5, 2021 at 7:51 PM Peter Geoghegan wrote: > Here are some specific checks I have in mind: One more for the list: * Validate PageIsAllVisible() for each page. In other words, pg_visibility should be merged with verify_heapam.c (or at least pg_visibility 's pg_check_fro

Re: amcheck's verify_heapam(), and HOT chain verification

2021-11-06 Thread Peter Geoghegan
ttle like a "mini index" stored on a heap page. Sequential scans don't care about LP_REDIRECTs at all. But index scans expect a great deal from LP_REDIRECTs. [1] https://www.postgresql.org/message-id/cah2-wznm4rcrhfaiwkpwtpew2bxdtgrozk7jwwgucxeh3d1...@mail.gmail.com -- Peter Geoghegan

Re: Draft release notes for next week's releases

2021-11-06 Thread Peter Geoghegan
On Fri, Nov 5, 2021 at 5:27 PM Tom Lane wrote: > As usual, please send any comments/corrections by Sunday. You probably won't miss this, but just in case: Alexander's very recent "Reset lastOverflowedXid on standby when needed" commit needs to be listed now. -- Peter Geoghegan

amcheck's verify_heapam(), and HOT chain verification

2021-11-05 Thread Peter Geoghegan
ing consistently gets it right. -- Peter Geoghegan

Re: parallel vacuum comments

2021-11-04 Thread Peter Geoghegan
On Thu, Nov 4, 2021 at 12:42 PM Peter Geoghegan wrote: > Since "The leader process alone processes all parallel-safe indexes in > the case where no workers are launched" (no change there), I wonder: > how does the leader *know* that it's the leader (and so can always > proce

<    7   8   9   10   11   12   13   14   15   16   >