On Wed, Jul 31, 2024 at 4:38 PM Peter Geoghegan wrote:
>
> On Thu, Jun 20, 2024 at 7:42 PM Melanie Plageman
> wrote:
> > In back branches starting with 14, failing to remove tuples older than
> > OldestXmin during pruning caused vacuum to infinitely loop in
> > lazy_scan_prune(), as investigated
On Thu, Jun 20, 2024 at 7:42 PM Melanie Plageman
wrote:
> In back branches starting with 14, failing to remove tuples older than
> OldestXmin during pruning caused vacuum to infinitely loop in
> lazy_scan_prune(), as investigated on this [1] thread.
Shouldn't somebody remove the entry that we
On Fri, Jul 26, 2024 at 1:27 PM Melanie Plageman
wrote:
>
> On Mon, Jul 22, 2024 at 9:26 PM Masahiko Sawada wrote:
> >
> > + CREATE TABLE ${table1}(col1 int)
> > + WITH (autovacuum_enabled=false, fillfactor=10);
> > + INSERT INTO $table1 VALUES(7);
> > + INSERT INTO $table1 SELECT
On Saturday, July 27, 2024, Melanie Plageman
wrote:
>
>
> Yes, the only thing that is important is having two rounds of index
> vacuuming and having one tuple with a value matching my cursor
> condition before the first index vacuum and one after. What do you
> mean update only the last few
On Wed, Jul 24, 2024 at 3:43 AM John Naylor wrote:
>
> On Wed, Jul 24, 2024 at 5:40 AM Masahiko Sawada wrote:
>
> > Without MEMORY_CONTEXT_CHECK, if size is 16 bytes, required_size is
> > also 16 bytes as it's already 8-byte aligned and Bump_CHUNKHDRSZ is 0.
> > On the other hand with
On Wed, Jul 24, 2024 at 8:19 AM John Naylor wrote:
>
> On Wed, Jul 24, 2024 at 2:42 PM John Naylor wrote:
> > As for lowering the limit, we've experimented with 256kB here:
> >
> > https://www.postgresql.org/message-id/canwcazzutvz3lsypauyqvzcezxz7qe+9ntnhgyzdtwxpul+...@mail.gmail.com
> >
> > As
On Mon, Jul 22, 2024 at 9:26 PM Masahiko Sawada wrote:
>
> + CREATE TABLE ${table1}(col1 int)
> + WITH (autovacuum_enabled=false, fillfactor=10);
> + INSERT INTO $table1 VALUES(7);
> + INSERT INTO $table1 SELECT generate_series(1, $nrows) % 3;
> + CREATE INDEX on ${table1}(col1);
>
On Wed, Jul 24, 2024 at 2:42 PM John Naylor wrote:
> As for lowering the limit, we've experimented with 256kB here:
>
> https://www.postgresql.org/message-id/canwcazzutvz3lsypauyqvzcezxz7qe+9ntnhgyzdtwxpul+...@mail.gmail.com
>
> As I mention there, going lower than that would need a small amount
On Wed, Jul 24, 2024 at 5:40 AM Masahiko Sawada wrote:
> Without MEMORY_CONTEXT_CHECK, if size is 16 bytes, required_size is
> also 16 bytes as it's already 8-byte aligned and Bump_CHUNKHDRSZ is 0.
> On the other hand with MEMORY_CONTEXT_CHECK, the requied_size is
> bumped to 40 bytes as
On Tue, Jul 23, 2024 at 5:43 AM Melanie Plageman
wrote:
>
> On Mon, Jul 22, 2024 at 10:54 PM Masahiko Sawada
> wrote:
> >
> > On Mon, Jul 22, 2024 at 6:26 PM Melanie Plageman
> > wrote:
> > >
> > > On Mon, Jul 22, 2024 at 6:36 PM Tom Lane wrote:
> > > >
> > > > Melanie Plageman writes:
> > >
On Mon, Jul 22, 2024 at 10:54 PM Masahiko Sawada wrote:
>
> On Mon, Jul 22, 2024 at 6:26 PM Melanie Plageman
> wrote:
> >
> > On Mon, Jul 22, 2024 at 6:36 PM Tom Lane wrote:
> > >
> > > Melanie Plageman writes:
> > > > We've only run tests with this commit on some of the back branches for
> >
On Mon, Jul 22, 2024 at 6:26 PM Melanie Plageman
wrote:
>
> On Mon, Jul 22, 2024 at 6:36 PM Tom Lane wrote:
> >
> > Melanie Plageman writes:
> > > We've only run tests with this commit on some of the back branches for
> > > some of these animals. Of those, I don't see any failures so far. So,
>
On Mon, Jul 22, 2024 at 6:36 PM Tom Lane wrote:
>
> Melanie Plageman writes:
> > We've only run tests with this commit on some of the back branches for
> > some of these animals. Of those, I don't see any failures so far. So,
> > it seems the test instability is just related to trying to get
> >
On Mon, Jul 22, 2024 at 2:04 PM Melanie Plageman
wrote:
>
> On Mon, Jul 22, 2024 at 2:17 PM Melanie Plageman
> wrote:
> >
> > So, I've just gone through all the test failures on master and 17 for
> > mamba, gull, mereswine, and copperhead. I wanted to confirm that the
> > test was always failing
Michael Paquier writes:
> Sawada-san and John are the two ones in the best position to answer
> that. I'm not sure either how to force a second index pass, either.
Yeah, I think we've established that having some way to force that,
without using a huge test case, would be really desirable.
On Mon, Jul 22, 2024 at 10:37:17AM -0700, Andres Freund wrote:
> On 2024-07-22 13:16:49 -0400, Robert Haas wrote:
>> Laziness would have been not bothering to develop a TAP test for this
>> at all. Going to the trouble of creating one and not being able to
>> make it as fast or as stable as
Melanie Plageman writes:
> We've only run tests with this commit on some of the back branches for
> some of these animals. Of those, I don't see any failures so far. So,
> it seems the test instability is just related to trying to get
> multiple passes of index vacuuming reliably with TIDStore.
On Mon, Jul 22, 2024 at 2:17 PM Melanie Plageman
wrote:
>
> So, I've just gone through all the test failures on master and 17 for
> mamba, gull, mereswine, and copperhead. I wanted to confirm that the
> test was always failing for the same reason and also if it had any
> failures pre-TIDStore.
>
On Mon, Jul 22, 2024 at 2:13 PM Andres Freund wrote:
> It's hard by now (i.e. 17+) because you need substantial amounts of rows to be
> able to trigger it which makes it a hard fight to introduce.
I didn't think that it was particularly hard when I tested the test
that Melanie committed.
> And
On Mon, Jul 22, 2024 at 1:17 PM Robert Haas wrote:
>
> On Mon, Jul 22, 2024 at 11:48 AM Tom Lane wrote:
> > I'm a little suspicious
> > of using it for tests that merely take an unreasonable amount of
> > time --- to me, that indicates laziness on the part of the test
> > author.
>
> Laziness
On 2024-07-22 12:00:51 -0400, Peter Geoghegan wrote:
> On Mon, Jul 22, 2024 at 11:49 AM Tom Lane wrote:
> > >> Andres has suggested in the past that we allow maintenance_work_mem be
> > >> set to a lower value or introduce some kind of development GUC so that
> > >> we can more easily test
On 2024-07-22 13:16:49 -0400, Robert Haas wrote:
> On Mon, Jul 22, 2024 at 11:48 AM Tom Lane wrote:
> > I'm a little suspicious
> > of using it for tests that merely take an unreasonable amount of
> > time --- to me, that indicates laziness on the part of the test
> > author.
>
> Laziness would
On Mon, Jul 22, 2024 at 11:48 AM Tom Lane wrote:
> I'm a little suspicious
> of using it for tests that merely take an unreasonable amount of
> time --- to me, that indicates laziness on the part of the test
> author.
Laziness would have been not bothering to develop a TAP test for this
at all.
Hi,
On 2024-07-21 12:51:51 -0400, Tom Lane wrote:
> Melanie Plageman writes:
> > When I run it on my machine with some added logging, the space taken
> > by dead items is about 330 kB more than maintenance_work_mem (which is
> > set to 1 MB). I could roughly double the excess by increasing the
>
On Mon, Jul 22, 2024 at 11:49 AM Tom Lane wrote:
> >> Andres has suggested in the past that we allow maintenance_work_mem be
> >> set to a lower value or introduce some kind of development GUC so that
> >> we can more easily test multiple pass index vacuuming. Do you think
> >> this would be
Alvaro Herrera writes:
> I think it's a bad idea to require buildfarm owners to edit their config
> files as we add tests that depend on PG_TEST_EXTRA. AFAIR we invented
> that setting so that tests that had security implications could be made
> opt-in instead of opt-out; I think this was a
Peter Geoghegan writes:
> On Mon, Jul 22, 2024 at 9:32 AM Melanie Plageman
> wrote:
>> Andres has suggested in the past that we allow maintenance_work_mem be
>> set to a lower value or introduce some kind of development GUC so that
>> we can more easily test multiple pass index vacuuming. Do you
Melanie Plageman writes:
> On Sun, Jul 21, 2024 at 5:04 PM Tom Lane wrote:
>> I note also that the PG_TEST_EXTRA approach has caused xid_wraparound
>> to get next-to-zero buildfarm coverage. If that test is actually
>> capable of revealing problems, we're unlikely to find out under the
>>
On Mon, Jul 22, 2024 at 9:32 AM Melanie Plageman
wrote:
> Andres has suggested in the past that we allow maintenance_work_mem be
> set to a lower value or introduce some kind of development GUC so that
> we can more easily test multiple pass index vacuuming. Do you think
> this would be worth it?
On 2024-Jul-22, Melanie Plageman wrote:
> On Sun, Jul 21, 2024 at 5:04 PM Tom Lane wrote:
> > I note also that the PG_TEST_EXTRA approach has caused xid_wraparound
> > to get next-to-zero buildfarm coverage. If that test is actually
> > capable of revealing problems, we're unlikely to find out
On Sun, Jul 21, 2024 at 4:29 PM Peter Geoghegan wrote:
>
> On Sun, Jul 21, 2024 at 12:51 PM Tom Lane wrote:
> > I do not think the answer to this is to nag the respective animal
> > owners to raise PG_TEST_TIMEOUT_DEFAULT. IMV this test is simply
> > not worth the cycles it takes, at least not
On Sun, Jul 21, 2024 at 5:04 PM Tom Lane wrote:
>
> Peter Geoghegan writes:
> > On Sun, Jul 21, 2024 at 12:51 PM Tom Lane wrote:
> >> I do not think the answer to this is to nag the respective animal
> >> owners to raise PG_TEST_TIMEOUT_DEFAULT. IMV this test is simply
> >> not worth the
On Sun, Jul 21, 2024 at 12:51 PM Tom Lane wrote:
>
> Melanie Plageman writes:
> > When I run it on my machine with some added logging, the space taken
> > by dead items is about 330 kB more than maintenance_work_mem (which is
> > set to 1 MB). I could roughly double the excess by increasing the
On Sun, Jul 21, 2024 at 5:04 PM Tom Lane wrote:
> > There will always be a small number of extremely slow buildfarm
> > animals. Optimizing for things like Raspberry pi animals with SD cards
> > just doesn't seem like a good use of developer time. I really care
> > about keeping the tests fast,
Peter Geoghegan writes:
> On Sun, Jul 21, 2024 at 12:51 PM Tom Lane wrote:
>> I do not think the answer to this is to nag the respective animal
>> owners to raise PG_TEST_TIMEOUT_DEFAULT. IMV this test is simply
>> not worth the cycles it takes, at least not for these machines.
> Can't we just
On Sun, Jul 21, 2024 at 12:51 PM Tom Lane wrote:
> I do not think the answer to this is to nag the respective animal
> owners to raise PG_TEST_TIMEOUT_DEFAULT. IMV this test is simply
> not worth the cycles it takes, at least not for these machines.
Can't we just move it to PG_TEST_EXTRA?
Melanie Plageman writes:
> When I run it on my machine with some added logging, the space taken
> by dead items is about 330 kB more than maintenance_work_mem (which is
> set to 1 MB). I could roughly double the excess by increasing the
> number of inserted tuples from 40 to 60. I'll do
On Wed, Jul 17, 2024 at 12:11 PM Peter Geoghegan wrote:
>
> On Wed, Jul 17, 2024 at 12:07 PM Melanie Plageman
> wrote:
> > We didn't end up doing two index vacuum passes. Because it doesn't
> > repro locally for me, I can only assume that the conditions for
> > forcing two index vacuuming passes
On Wed, Jul 17, 2024 at 12:07 PM Melanie Plageman
wrote:
> We didn't end up doing two index vacuum passes. Because it doesn't
> repro locally for me, I can only assume that the conditions for
> forcing two index vacuuming passes in master just weren't met in this
> case. I'm unsurprised, as it is
On Wed, Jul 17, 2024 at 11:07 AM Melanie Plageman
wrote:
>
> On Mon, Jul 15, 2024 at 6:02 PM Peter Geoghegan wrote:
> >
> > I think that there is some chance that this test will break the build
> > farm in whatever way, since there is a long history of VACUUM not
> > quite behaving as expected
On Mon, Jul 15, 2024 at 6:02 PM Peter Geoghegan wrote:
>
> On Mon, Jul 8, 2024 at 2:25 PM Melanie Plageman
> wrote:
> > I could still use another pair of eyes on the test (looking out for
> > stability enhancing measures I could take).
>
> First, the basics: I found that your test failed
On Mon, Jul 8, 2024 at 2:25 PM Melanie Plageman
wrote:
> Attached v3 has one important additional component in the test -- I
> use pg_stat_progress_vacuum to confirm that we actually do more than
> one pass of index vacuuming. Otherwise, it would have been trivial for
> the test to incorrectly
On Tue, Jul 2, 2024 at 7:07 PM Melanie Plageman
wrote:
>
> On Mon, Jun 24, 2024 at 4:27 AM Heikki Linnakangas wrote:
> >
> > Would it be possible to make it robust so that we could always run it
> > with "make check"? This seems like an important corner case to
> > regression test.
>
> Okay,
On Mon, Jun 24, 2024 at 4:27 AM Heikki Linnakangas wrote:
>
> Would it be possible to make it robust so that we could always run it
> with "make check"? This seems like an important corner case to
> regression test.
Okay, I've attached a new version of the patch and a new version of
the repro
On Tue, Jun 25, 2024 at 4:41 PM Andres Freund wrote:
> I doubt that's doable in the back branches. And even on HEAD, I don't think
> it's a particularly attractive option - there's just a global vistest for each
> of the types of objects with a specific horizon (they need to be updated
>
On 2024-06-25 14:35:00 -0400, Robert Haas wrote:
> Is there any way that we could instead tweak things so that we adjust
> the visibility test object itself? Like can have a GlobalVisTest API
> where we can supply the OldestXmin from the VacuumCutoffs and have it
> ... do something useful with
On Tue, Jun 25, 2024 at 1:10 PM Andres Freund wrote:
> That said, obviously there will be plenty setups where this won't cause an
> issue. I don't really have a handle on how often it'd be a problem.
Fair enough. Even if it's not super-common, it doesn't seem like a
great idea to regress such
Hi,
On 2024-06-25 12:31:11 -0400, Robert Haas wrote:
> On Tue, Jun 25, 2024 at 11:39 AM Melanie Plageman
> wrote:
> > On Tue, Jun 25, 2024 at 10:31 AM Robert Haas wrote:
> > > On Tue, Jun 25, 2024 at 9:07 AM Andres Freund wrote:
> > > > It's not hard - but it has downsides. It'll mean that -
On Tue, Jun 25, 2024 at 11:39 AM Melanie Plageman
wrote:
> On Tue, Jun 25, 2024 at 10:31 AM Robert Haas wrote:
> > On Tue, Jun 25, 2024 at 9:07 AM Andres Freund wrote:
> > > It's not hard - but it has downsides. It'll mean that - outside of vacuum
> > > -
> > > we'll much more often not react
On Tue, Jun 25, 2024 at 10:31 AM Robert Haas wrote:
>
> On Tue, Jun 25, 2024 at 9:07 AM Andres Freund wrote:
> > It's not hard - but it has downsides. It'll mean that - outside of vacuum -
> > we'll much more often not react to horizons going backwards due to
> > hot_standby_feedback. Which
On 24.06.2024 17:37, Melanie Plageman wrote:
On Mon, Jun 24, 2024 at 4:10 AM Alena Rybakina wrote:
We can fix this by always removing tuples considered dead before
VacuumCutoffs->OldestXmin. This is okay even if a reconnected standby
has a transaction that sees that tuple as alive, because it
On Tue, Jun 25, 2024 at 9:07 AM Andres Freund wrote:
> It's not hard - but it has downsides. It'll mean that - outside of vacuum -
> we'll much more often not react to horizons going backwards due to
> hot_standby_feedback. Which means that hot_standby_feedback, when used without
> slots, will
On 2024-06-25 08:42:02 -0400, Robert Haas wrote:
> On Tue, Jun 25, 2024 at 8:03 AM Andres Freund wrote:
> > I think that's going in the wrong direction. We *want* to prune more
> > aggressively if we can (*), the necessary state is represented by the
> > vistest. That's a different thing than
On Tue, Jun 25, 2024 at 8:03 AM Andres Freund wrote:
> I think that's going in the wrong direction. We *want* to prune more
> aggressively if we can (*), the necessary state is represented by the
> vistest. That's a different thing than *having* to prune tuples beyond a
> certain xmin (the cutoff
Hi,
On 2024-06-24 16:35:50 -0400, Robert Haas wrote:
> On Mon, Jun 24, 2024 at 3:23 PM Melanie Plageman
> wrote:
> > Are you more concerned about having a single horizon for pruning or
> > about having a horizon that does not move backwards after being
> > established at the beginning of
On Mon, Jun 24, 2024 at 09:49:53PM -0400, Peter Geoghegan wrote:
> On Mon, Jun 24, 2024 at 9:30 PM Noah Misch wrote:
> > On Mon, Jun 24, 2024 at 03:23:39PM -0400, Melanie Plageman wrote:
> > > Right now, in master, we do use a single horizon when determining what
> > > is pruned -- that from
On Mon, Jun 24, 2024 at 9:30 PM Noah Misch wrote:
> On Mon, Jun 24, 2024 at 03:23:39PM -0400, Melanie Plageman wrote:
> > Right now, in master, we do use a single horizon when determining what
> > is pruned -- that from GlobalVisState. OldestXmin is only used for
> > freezing and full page
On Mon, Jun 24, 2024 at 04:51:24PM -0400, Peter Geoghegan wrote:
> On Mon, Jun 24, 2024 at 4:36 PM Robert Haas wrote:
> > I'm not sure I understand. The most important thing here is fixing the
> > bug. But if we have a choice of how to fix the bug, I'd prefer to do
> > it by having the pruning
On Mon, Jun 24, 2024 at 4:51 PM Peter Geoghegan wrote:
>
> On Mon, Jun 24, 2024 at 4:36 PM Robert Haas wrote:
> > I thought the idea was that the GlobalVisTest stuff would force a
> > recalculation now and then, but maybe it doesn't actually do that?
>
> It definitely can do that. Just not in a
On Mon, Jun 24, 2024 at 4:36 PM Robert Haas wrote:
> I'm not sure I understand. The most important thing here is fixing the
> bug. But if we have a choice of how to fix the bug, I'd prefer to do
> it by having the pruning code test one horizon that is always correct,
> rather than (as I think the
On Mon, Jun 24, 2024 at 4:42 PM Peter Geoghegan wrote:
>
> On Mon, Jun 24, 2024 at 3:36 PM Melanie Plageman
> wrote:
> > One thing I don't understand is why it is okay to freeze the xmax of a
> > dead tuple just because it is from an aborted update.
>
> We don't do that with XID-based xmaxs.
On Mon, Jun 24, 2024 at 3:36 PM Melanie Plageman
wrote:
> One thing I don't understand is why it is okay to freeze the xmax of a
> dead tuple just because it is from an aborted update.
We don't do that with XID-based xmaxs. Though perhaps we should, since
we'll already prune-away the successor
On Mon, Jun 24, 2024 at 3:23 PM Melanie Plageman
wrote:
> Are you more concerned about having a single horizon for pruning or
> about having a horizon that does not move backwards after being
> established at the beginning of vacuuming the relation?
I'm not sure I understand. The most important
On Thu, Jun 20, 2024 at 7:42 PM Melanie Plageman
wrote:
>
> If vacuum fails to remove a tuple with xmax older than
> VacuumCutoffs->OldestXmin and younger than
> GlobalVisState->maybe_needed, it will ERROR out when determining
> whether or not to freeze the tuple with "cannot freeze committed
>
On Mon, Jun 24, 2024 at 3:23 PM Melanie Plageman
wrote:
> I had always thought it was because the vacuuming backend's
> GlobalVisState will get updated periodically throughout vacuum and so,
> assuming the oldest running transaction changes, our horizon for
> vacuum would change.
I believe that
On Mon, Jun 24, 2024 at 1:05 PM Robert Haas wrote:
>
> On Mon, Jun 24, 2024 at 12:43 PM Peter Geoghegan wrote:
> > The problem here is that OldestXmin is supposed to be more
> > conservative than vistest, which it almost always is, except in this
> > one edge case. I don't think that plugging
On Mon, Jun 24, 2024 at 1:05 PM Robert Haas wrote:
> On Mon, Jun 24, 2024 at 12:43 PM Peter Geoghegan wrote:
> > The problem here is that OldestXmin is supposed to be more
> > conservative than vistest, which it almost always is, except in this
> > one edge case. I don't think that plugging that
On Mon, Jun 24, 2024 at 12:43 PM Peter Geoghegan wrote:
> The problem here is that OldestXmin is supposed to be more
> conservative than vistest, which it almost always is, except in this
> one edge case. I don't think that plugging that hole changes the basic
> fact that there is one source of
On Mon, Jun 24, 2024 at 11:44 AM Robert Haas wrote:
> I don't have a great feeling about this fix. It's not that I think
> it's wrong. It's just that the underlying problem here is that we have
> heap_page_prune_and_freeze() getting both GlobalVisState *vistest and
> struct VacuumCutoffs
On Thu, Jun 20, 2024 at 7:42 PM Melanie Plageman
wrote:
> We can fix this by always removing tuples considered dead before
> VacuumCutoffs->OldestXmin.
I don't have a great feeling about this fix. It's not that I think
it's wrong. It's just that the underlying problem here is that we have
On 6/24/24 16:53, Melanie Plageman wrote:
> On Mon, Jun 24, 2024 at 4:27 AM Heikki Linnakangas wrote:
>>
>> On 21/06/2024 03:02, Peter Geoghegan wrote:
>>> On Thu, Jun 20, 2024 at 7:42 PM Melanie Plageman
>>> wrote:
>>>
The repro forces a round of index vacuuming after the standby
On Mon, Jun 24, 2024 at 4:27 AM Heikki Linnakangas wrote:
>
> On 21/06/2024 03:02, Peter Geoghegan wrote:
> > On Thu, Jun 20, 2024 at 7:42 PM Melanie Plageman
> > wrote:
> >
> >> The repro forces a round of index vacuuming after the standby
> >> reconnects and before pruning a dead tuple whose
On Mon, Jun 24, 2024 at 4:10 AM Alena Rybakina wrote:
>
> We can fix this by always removing tuples considered dead before
> VacuumCutoffs->OldestXmin. This is okay even if a reconnected standby
> has a transaction that sees that tuple as alive, because it will
> simply wait to replay the removal
On 21/06/2024 03:02, Peter Geoghegan wrote:
On Thu, Jun 20, 2024 at 7:42 PM Melanie Plageman
wrote:
If vacuum fails to remove a tuple with xmax older than
VacuumCutoffs->OldestXmin and younger than
GlobalVisState->maybe_needed, it will ERROR out when determining
whether or not to freeze the
Hi, Melanie! I'm glad to hear you that you have found a root case of the
problem) Thank you for that!
On 21.06.2024 02:42, Melanie Plageman wrote:
Hi,
If vacuum fails to remove a tuple with xmax older than
VacuumCutoffs->OldestXmin and younger than
GlobalVisState->maybe_needed, it will ERROR
On Thu, Jun 20, 2024 at 7:42 PM Melanie Plageman
wrote:
> If vacuum fails to remove a tuple with xmax older than
> VacuumCutoffs->OldestXmin and younger than
> GlobalVisState->maybe_needed, it will ERROR out when determining
> whether or not to freeze the tuple with "cannot freeze committed
>
Hi,
If vacuum fails to remove a tuple with xmax older than
VacuumCutoffs->OldestXmin and younger than
GlobalVisState->maybe_needed, it will ERROR out when determining
whether or not to freeze the tuple with "cannot freeze committed
xmax".
In back branches starting with 14, failing to remove
77 matches
Mail list logo
