On Tue, Dec 15, 2020 at 6:44 PM Masahiko Sawada wrote:
> In connection with this change, we would need to rethink the meaning
> of the INDEX_CLEANUP option. As of now, if it's not set (i.g.
> VACOPT_TERNARY_DEFAULT in the code), it's treated as true and will do
> heap clean. But I think we can mak
On Sat, Nov 21, 2020 at 8:03 AM Peter Geoghegan wrote:
>
> On Fri, Nov 20, 2020 at 2:17 PM Robert Haas wrote:
> > > Does that make sense?
> >
> > I *think* so. For me the point is that the index never has a right to
> > insist on being vacuumed, but it can offer an opinion on how helpful
> > it w
On Fri, Nov 20, 2020 at 2:17 PM Robert Haas wrote:
> > Does that make sense?
>
> I *think* so. For me the point is that the index never has a right to
> insist on being vacuumed, but it can offer an opinion on how helpful
> it would be.
Right, that might be the single most important point. It's a
On Fri, Nov 20, 2020 at 4:21 PM Peter Geoghegan wrote:
> Actually, now I think that BRIN shouldn't be special to vacuumlazy.c
> in any way. It doesn't make sense as part of this future world in
> which index vacuuming can be skipped for individual indexes (which is
> what I talked to Sawada-san ab
On Fri, Nov 20, 2020 at 12:04 PM Robert Haas wrote:
> That's an interesting idea. We should think about the needs of brin
> indexes when designing something better than the current system. They
> have the interesting property that the heap deciding to change LP_DEAD
> to LP_UNUSED doesn't break an
On Fri, Nov 20, 2020 at 2:37 PM Peter Geoghegan wrote:
> Right. Maybe we don't ask the index AMs for discrete yes/no answers.
> Maybe we can ask them for a continuous answer, such as a value between
> 0.0 and 1.0 that represents the urgency/bloat, or something like that.
> And so the final yes/no
On Fri, Nov 20, 2020 at 3:17 AM Masahiko Sawada wrote:
> I had missed your bottom-up index deletion patch but it's a promising
> improvement. With that patch, the number of dead tuples in individual
> indexes may differ. So it's important that we make index vacuuming a
> per-index thing.
Right, b
On Thu, Nov 19, 2020 at 8:58 PM Masahiko Sawada wrote:
> For HEAD, there was a discussion that we change lazy vacuum and
> bulkdelete and vacuumcleanup APIs so that it calls these APIs even
> when INDEX_CLEANUP is specified. That is, when INDEX_CLEANUP false is
> specified, it collects dead tuple
On Fri, Nov 20, 2020 at 12:56 PM Peter Geoghegan wrote:
>
> On Thu, Nov 19, 2020 at 5:58 PM Masahiko Sawada wrote:
> > Several months passed from the discussion. We decided not to do
> > anything on back branches but IIUC the fundamental issue is not fixed
> > yet. The issue pointed out by Andres
On Thu, Nov 19, 2020 at 5:58 PM Masahiko Sawada wrote:
> Several months passed from the discussion. We decided not to do
> anything on back branches but IIUC the fundamental issue is not fixed
> yet. The issue pointed out by Andres that we should leave the index AM
> to decide whether doing vacuum
On Mon, Jun 29, 2020 at 9:51 PM Masahiko Sawada
wrote:
>
> On Sun, 28 Jun 2020 at 02:44, Peter Geoghegan wrote:
> >
> > On Fri, Jun 26, 2020 at 10:15 PM Masahiko Sawada
> > wrote:
> > > Regarding to the extent of the impact, this bug will affect the user
> > > who turned vacuum_index_cleanup off
On Sun, 28 Jun 2020 at 02:44, Peter Geoghegan wrote:
>
> On Fri, Jun 26, 2020 at 10:15 PM Masahiko Sawada
> wrote:
> > Regarding to the extent of the impact, this bug will affect the user
> > who turned vacuum_index_cleanup off or executed manually vacuum with
> > INDEX_CLEANUP off for a long tim
On Fri, Jun 26, 2020 at 10:15 PM Masahiko Sawada
wrote:
> Regarding to the extent of the impact, this bug will affect the user
> who turned vacuum_index_cleanup off or executed manually vacuum with
> INDEX_CLEANUP off for a long time, after some vacuums. On the other
> hand, the user who uses INDE
On Sat, 27 Jun 2020 at 08:00, Peter Geoghegan wrote:
>
> On Fri, Jun 26, 2020 at 5:39 AM Robert Haas wrote:
> > My opinion is that there's no need to change the code in the
> > back-branches, and that I don't really like the approach in master
> > either.
>
> I guess it's hard to see a way that w
On Fri, Jun 26, 2020 at 5:39 AM Robert Haas wrote:
> My opinion is that there's no need to change the code in the
> back-branches, and that I don't really like the approach in master
> either.
I guess it's hard to see a way that we could fix this in the
backbranches, provided we aren't willing to
On Thu, Jun 25, 2020 at 8:44 PM Peter Geoghegan wrote:
> I am sure about this much: The design embodied by Masahiko's patch is
> clearly a better one overall, even if it doesn't fix the problem on
> its own. I agree that we cannot allow nbtree to ignore "INDEX_CLEANUP
> = off", even if that means
On Thu, Jun 25, 2020 at 8:28 AM Robert Haas wrote:
> I'm not really convinced. I agree that from a theoretical point of
> view an index can have arbitrary needs and is the arbiter of its own
> needs, but when I pull the emergency break, I want the vehicle to
> stop, not think about stopping.
Maki
On Thu, Jun 25, 2020 at 6:59 AM Masahiko Sawada
wrote:
> I think that with the approach implemented in my patch, it could be a
> problem for the user that the user cannot easily know in advance
> whether vacuum with INDEX_CLEANUP false will perform index cleanup,
> even if page deletion doesn’t ha
On Wed, Jun 24, 2020 at 4:02 PM Peter Geoghegan wrote:
> > Apparently, what we're really doing here is that even when
> > INDEX_CLEANUP is OFF, we're still going to keep all the dead tuples.
> > That seems sad, but if it's what we have to do then it at least needs
> > comments explaining it.
>
> +
On Thu, 25 Jun 2020 at 05:02, Peter Geoghegan wrote:
>
> On Wed, Jun 24, 2020 at 10:21 AM Robert Haas wrote:
> > Sorry, I'm so far behind on my email. Argh.
>
> That's okay.
>
> > I think, especially on the blog post you linked, that we should aim to
> > have INDEX_CLEANUP OFF mode do the minimum
On Wed, Jun 24, 2020 at 10:21 AM Robert Haas wrote:
> Sorry, I'm so far behind on my email. Argh.
That's okay.
> I think, especially on the blog post you linked, that we should aim to
> have INDEX_CLEANUP OFF mode do the minimum possible amount of work
> while still keeping us safe against trans
On Fri, May 22, 2020 at 4:40 PM Peter Geoghegan wrote:
> On Mon, May 18, 2020 at 7:32 PM Masahiko Sawada
> wrote:
> > I've attached WIP patch for HEAD. With this patch, the core pass
> > index_cleanup to bulkdelete and vacuumcleanup callbacks so that they
> > can make decision whether run vacuum
On Tue, 19 May 2020 at 11:32, Masahiko Sawada
wrote:
>
> On Thu, 7 May 2020 at 16:26, Masahiko Sawada
> wrote:
> >
> > On Thu, 7 May 2020 at 15:40, Masahiko Sawada
> > wrote:
> > >
> > > On Thu, 7 May 2020 at 03:28, Peter Geoghegan wrote:
> > > >
> > > > On Wed, May 6, 2020 at 2:28 AM Masahiko
On Fri, May 22, 2020 at 1:40 PM Peter Geoghegan wrote:
> It. seems like the right direction to me. Robert?
BTW, this is an interesting report of somebody using the INDEX_CLEANUP
feature when they had to deal with a difficult production issue:
https://www.buildkitestatus.com/incidents/h0vnx4gp7dj
On Mon, May 18, 2020 at 7:32 PM Masahiko Sawada
wrote:
> I've attached WIP patch for HEAD. With this patch, the core pass
> index_cleanup to bulkdelete and vacuumcleanup callbacks so that they
> can make decision whether run vacuum or not.
>
> If the direction of this patch seems good, I'll change
On Thu, 7 May 2020 at 16:26, Masahiko Sawada
wrote:
>
> On Thu, 7 May 2020 at 15:40, Masahiko Sawada
> wrote:
> >
> > On Thu, 7 May 2020 at 03:28, Peter Geoghegan wrote:
> > >
> > > On Wed, May 6, 2020 at 2:28 AM Masahiko Sawada
> > > wrote:
> > > > I've attached the patch fixes this issue.
> >
On Thu, 7 May 2020 at 15:40, Masahiko Sawada
wrote:
>
> On Thu, 7 May 2020 at 03:28, Peter Geoghegan wrote:
> >
> > On Wed, May 6, 2020 at 2:28 AM Masahiko Sawada
> > wrote:
> > > I've attached the patch fixes this issue.
> > >
> > > With this patch, we don't skip only index cleanup phase when
>
On Thu, 7 May 2020 at 03:28, Peter Geoghegan wrote:
>
> On Wed, May 6, 2020 at 2:28 AM Masahiko Sawada
> wrote:
> > I've attached the patch fixes this issue.
> >
> > With this patch, we don't skip only index cleanup phase when
> > performing an aggressive vacuum. The reason why I don't skip only
On Wed, May 6, 2020 at 1:06 PM Alvaro Herrera wrote:
> Good question. I agree that BRIN summarization is not at all related to
> what other index AMs do during the cleanup phase. However, if the
> index_cleanup feature is there to make it faster to process a table
> that's nearing wraparound haz
On 2020-May-06, Peter Geoghegan wrote:
> Also, do we really want to skip summarization of BRIN indexes? This
> cleanup is rather dissimilar to the cleanup that takes place in most
> other AMs -- it isn't really that related to garbage collection (BRIN
> is rather unique in this respect). I think t
On Wed, May 6, 2020 at 11:28 AM Peter Geoghegan wrote:
> This approach has an obvious disadvantage: the patch really has to
> teach *every* index AM to do something with that state (most will
> simply do no work). It seems logical to have the index AM control what
> happens, though. This allows th
On Wed, May 6, 2020 at 2:28 AM Masahiko Sawada
wrote:
> I've attached the patch fixes this issue.
>
> With this patch, we don't skip only index cleanup phase when
> performing an aggressive vacuum. The reason why I don't skip only
> index cleanup phase is that index vacuum phase can be called mult
On Wed, 6 May 2020 at 07:17, Masahiko Sawada
wrote:
>
> On Wed, 6 May 2020 at 07:14, Peter Geoghegan wrote:
> >
> > On Tue, May 5, 2020 at 2:52 PM Masahiko Sawada
> > wrote:
> > > So IIUC the problem is that since we skip both,
> > > oldst_btpo_xact could be seen as a "future" xid during vacuum.
On Wed, 6 May 2020 at 07:14, Peter Geoghegan wrote:
>
> On Tue, May 5, 2020 at 2:52 PM Masahiko Sawada
> wrote:
> > So IIUC the problem is that since we skip both,
> > oldst_btpo_xact could be seen as a "future" xid during vacuum. Which
> > will be a cause of that vacuum misses pages which can ac
On Tue, May 5, 2020 at 2:52 PM Masahiko Sawada
wrote:
> So IIUC the problem is that since we skip both,
> oldst_btpo_xact could be seen as a "future" xid during vacuum. Which
> will be a cause of that vacuum misses pages which can actually be
> recycled.
This is also my understanding of the probl
On Sat, 18 Apr 2020 at 03:22, Robert Haas wrote:
>
> On Thu, Apr 16, 2020 at 6:49 PM Andres Freund wrote:
> > Yea. _bt_vacuum_needs_cleanup() needs to check if
> > metad->btm_oldest_btpo_xact is older than the FreezeLimit computed by
> > vacuum_set_xid_limits() and vacuum the index if so even if
On Wed, Apr 29, 2020 at 2:56 PM Andres Freund wrote:
> I'm not sure I see the advantage. Only doing so in the freezing case
> seems unlikely to cause additional conflicts, but I'm less sure about
> doing it always. btpo.xact will often be quite recent, right? So it'd
> likely cause more conflicts.
Hi,
On 2020-04-29 13:40:55 -0700, Peter Geoghegan wrote:
> On Wed, Apr 29, 2020 at 12:54 PM Andres Freund wrote:
> > > Fundamentally, btvacuumpage() doesn't freeze 32-bit XIDs (from
> > > bpto.xact) when it recycles deleted pages. It simply puts them in the
> > > FSM without changing anything abo
On Wed, Apr 29, 2020 at 2:04 PM Peter Geoghegan wrote:
> As a bonus, we now have an easy correctness cross-check: if some
> random piece of nbtree code lands on a page (having followed a
> downlink or sibling link) that is marked recycled, then clearly
> something is very wrong -- throw a "can't h
On Wed, Apr 29, 2020 at 1:40 PM Peter Geoghegan wrote:
> I'm not sure how low the costs would be, but at least we'd only have
> to do it once per already-deleted page (i.e. only the first time a
> VACUUM operation found _bt_page_eligible_for_recycling() returned true
> for the page and marked it r
On Wed, Apr 29, 2020 at 12:54 PM Andres Freund wrote:
> > Fundamentally, btvacuumpage() doesn't freeze 32-bit XIDs (from
> > bpto.xact) when it recycles deleted pages. It simply puts them in the
> > FSM without changing anything about the page itself. This means
> > surprisingly little in the cont
Hi,
On 2020-04-29 11:28:00 -0700, Peter Geoghegan wrote:
> On Thu, Apr 16, 2020 at 11:27 AM Andres Freund wrote:
> > Sure, there is some pre-existing wraparound danger for individual
> > pages. But it's a pretty narrow corner case before INDEX_CLEANUP
> > off.
> >
> > That comment says something
On Thu, Apr 16, 2020 at 11:27 AM Andres Freund wrote:
> Sure, there is some pre-existing wraparound danger for individual
> pages. But it's a pretty narrow corner case before INDEX_CLEANUP
> off.
>
> That comment says something about "shared-memory free space map", making
> it sound like any crash
On Wed, Apr 22, 2020 at 9:05 PM Peter Geoghegan wrote:
> > Given hint bits it seems fairly hard to make that a reliable check.
>
> I don't follow. It doesn't have to be a perfect check. Detecting if
> there is *any* buffer lock held at all would be a big improvement.
It is true that the assumptio
On Wed, Apr 22, 2020 at 8:33 PM Andres Freund wrote:
> On 2020-04-22 20:08:42 -0700, Peter Geoghegan wrote:
> > I can get Valgrind to complain about it when the regression tests are
> > run with the attached patch applied.
>
> Nice! Have you checked how much of an incremental slowdown this causes
Hi,
On 2020-04-22 20:08:42 -0700, Peter Geoghegan wrote:
> I can get Valgrind to complain about it when the regression tests are
> run with the attached patch applied.
Nice! Have you checked how much of an incremental slowdown this causes?
> This patch is very rough -- it was just the first th
On Wed, Apr 22, 2020 at 6:05 PM Peter Geoghegan wrote:
> (It would be nice if we could teach Valgrind to "poison" buffers when
> we don't have a pin held...that would probably have caught this issue
> almost immediately.)
I can get Valgrind to complain about it when the regression tests are
run w
On Thu, Apr 16, 2020 at 12:30 AM Masahiko Sawada wrote:
> For btree indexes, IIRC skipping index cleanup could not be a cause of
> corruption, but be a cause of index bloat since it leaves recyclable
> pages which are not marked as recyclable.
I spotted a bug in "Skip full index scan during clean
On Thu, Apr 16, 2020 at 6:49 PM Andres Freund wrote:
> Yea. _bt_vacuum_needs_cleanup() needs to check if
> metad->btm_oldest_btpo_xact is older than the FreezeLimit computed by
> vacuum_set_xid_limits() and vacuum the index if so even if INDEX_CLEANUP
> false.
I'm still fairly unclear on what the
On Fri, 17 Apr 2020 at 02:58, Andres Freund wrote:
>
> Hi,
>
> On 2020-04-16 16:30:02 +0900, Masahiko Sawada wrote:
> > For btree indexes, IIRC skipping index cleanup could not be a cause of
> > corruption, but be a cause of index bloat since it leaves recyclable
> > pages which are not marked as
On Thu, Apr 16, 2020 at 3:49 PM Andres Freund wrote:
> I think we really just stop being miserly and update the xid to be
> FrozenTransactionId or InvalidTransactionId when vacuum encounters one
> that's from before the the xid cutoff used by vacuum (i.e. what could
> become the new relfrozenxid).
Hi,
On 2020-04-16 13:28:00 -0700, Peter Geoghegan wrote:
> > And, what's worse, in the INDEX_CLEANUP off case, future VACUUMs with
> > INDEX_CLEANUP on might not even visit the index. As there very well
> > might not be many dead heap tuples around anymore (previous vacuums with
> > cleanup off wi
On Thu, Apr 16, 2020 at 11:27 AM Andres Freund wrote:
> Sure, there is some pre-existing wraparound danger for individual
> pages. But it's a pretty narrow corner case before INDEX_CLEANUP
> off.
It's a matter of degree. Hard to judge something like that.
> And, what's worse, in the INDEX_CLEANU
Hi,
On 2020-04-15 18:11:40 -0700, Peter Geoghegan wrote:
> On Wed, Apr 15, 2020 at 4:57 PM Robert Haas wrote:
> > I seem to recall Simon raising this issue at the time that the patch
> > was being discussed, and I thought that we had eventually decided that
> > it was OK for some reason. But I do
Hi,
On 2020-04-16 16:30:02 +0900, Masahiko Sawada wrote:
> For btree indexes, IIRC skipping index cleanup could not be a cause of
> corruption, but be a cause of index bloat since it leaves recyclable
> pages which are not marked as recyclable. The index bloat is the main
> side effect of skipping
On Thu, Apr 16, 2020 at 8:38 AM Andres Freund wrote:
>
> Hi,
>
> commit a96c41feec6b6616eb9d5baee9a9e08c20533c38
> Author: Robert Haas
> Date: 2019-04-04 14:58:53 -0400
>
> Allow VACUUM to be run with index cleanup disabled.
>
> This commit adds a new reloption, vacuum_index_cleanup, wh
On Wed, Apr 15, 2020 at 4:57 PM Robert Haas wrote:
> I seem to recall Simon raising this issue at the time that the patch
> was being discussed, and I thought that we had eventually decided that
> it was OK for some reason. But I don't remember the details, and it is
> possible that we got it wron
On Wed, Apr 15, 2020 at 7:38 PM Andres Freund wrote:
> It's possible that something protects against dangers in the case of
> INDEX CLEANUP false, or that the consequences aren't too bad. But I
> didn't see any comments about the danagers in the patch.
I seem to recall Simon raising this issue at
Hi,
commit a96c41feec6b6616eb9d5baee9a9e08c20533c38
Author: Robert Haas
Date: 2019-04-04 14:58:53 -0400
Allow VACUUM to be run with index cleanup disabled.
This commit adds a new reloption, vacuum_index_cleanup, which
controls whether index cleanup is performed for a particular
59 matches
Mail list logo
