Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-05-15 Thread Thomas Munro
On Thu, May 16, 2019 at 3:53 AM Tom Lane wrote: > Andres Freund writes: > > On 2019-05-15 12:01:07 +1200, Thomas Munro wrote: > >> This is listed as an open item to resolve for 12. IIUC the options on > >> the table are: > >> > >> 1. Do nothing, and ship with effective_io_concurrency + 10. >

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-05-15 Thread Tom Lane
Andres Freund writes: > On 2019-05-15 12:01:07 +1200, Thomas Munro wrote: >> This is listed as an open item to resolve for 12. IIUC the options on >> the table are: >> >> 1. Do nothing, and ship with effective_io_concurrency + 10. >> 2. Just use effective_io_concurrency without the hardwired

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-05-14 Thread Andres Freund
Hi, On 2019-05-15 12:01:07 +1200, Thomas Munro wrote: > On Thu, May 2, 2019 at 5:10 AM Robert Haas wrote: > > On Wed, May 1, 2019 at 12:50 PM Tom Lane wrote: > > > > Not strongly enough to argue about it very hard. The current behavior > > > > is a little weird, but it's a long way from being

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-05-14 Thread Thomas Munro
On Thu, May 2, 2019 at 5:10 AM Robert Haas wrote: > On Wed, May 1, 2019 at 12:50 PM Tom Lane wrote: > > > Not strongly enough to argue about it very hard. The current behavior > > > is a little weird, but it's a long way from being the weirdest thing > > > we ship, and it appears that we have

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-05-01 Thread Robert Haas
On Wed, May 1, 2019 at 12:50 PM Tom Lane wrote: > > Not strongly enough to argue about it very hard. The current behavior > > is a little weird, but it's a long way from being the weirdest thing > > we ship, and it appears that we have no tangible evidence that it > > causes a problem in

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-05-01 Thread Tom Lane
Robert Haas writes: > On Wed, May 1, 2019 at 12:15 PM Andres Freund wrote: >> My current inclination is to not do anything for v12. Robert, do you >> disagree? > Not strongly enough to argue about it very hard. The current behavior > is a little weird, but it's a long way from being the

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-05-01 Thread Robert Haas
On Wed, May 1, 2019 at 12:15 PM Andres Freund wrote: > On 2019-04-01 18:26:59 -0700, Andres Freund wrote: > > I'm not yet convinced it's necessary to create a new GUC, but also not > > strongly opposed. I've created an open items issue for it, so we don't > > forget. > > My current inclination

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-05-01 Thread Andres Freund
Hi, On 2019-04-01 18:26:59 -0700, Andres Freund wrote: > I'm not yet convinced it's necessary to create a new GUC, but also not > strongly opposed. I've created an open items issue for it, so we don't > forget. My current inclination is to not do anything for v12. Robert, do you disagree?

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-04-01 Thread Andres Freund
Hi, On 2019-03-30 11:44:36 -0400, Robert Haas wrote: > On Sat, Mar 30, 2019 at 6:33 AM Thomas Munro wrote: > > I didn't understand that last sentence. > > > > Here's an attempt to write a suitable comment for the quick fix. And > > I suppose effective_io_concurrency is a reasonable default. > >

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-04-01 Thread Simon Riggs
On Fri, 29 Mar 2019 at 16:32, Andres Freund wrote: > On 2019-03-29 16:20:54 +, Simon Riggs wrote: > > On Fri, 29 Mar 2019 at 16:12, Andres Freund wrote: > > > > > > > On 2019-03-29 15:58:14 +, Simon Riggs wrote: > > > > On Fri, 29 Mar 2019 at 15:29, Andres Freund > wrote: > > > > >

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-03-30 Thread Andres Freund
Hi, On March 30, 2019 5:33:12 PM EDT, Thomas Munro wrote: >On Sun, Mar 31, 2019 at 8:20 AM Peter Geoghegan wrote: >> On Sat, Mar 30, 2019 at 8:44 AM Robert Haas >wrote: >> > Overall I'm inclined to think that we're making the same mistake >here >> > that we did with work_mem, namely, assuming

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-03-30 Thread Thomas Munro
On Sun, Mar 31, 2019 at 8:20 AM Peter Geoghegan wrote: > On Sat, Mar 30, 2019 at 8:44 AM Robert Haas wrote: > > Overall I'm inclined to think that we're making the same mistake here > > that we did with work_mem, namely, assuming that you can control a > > bunch of different prefetching

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-03-30 Thread Peter Geoghegan
On Sat, Mar 30, 2019 at 8:44 AM Robert Haas wrote: > Overall I'm inclined to think that we're making the same mistake here > that we did with work_mem, namely, assuming that you can control a > bunch of different prefetching behaviors with a single GUC and things > will be OK. Let's just create

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-03-30 Thread Robert Haas
On Sat, Mar 30, 2019 at 6:33 AM Thomas Munro wrote: > I didn't understand that last sentence. > > Here's an attempt to write a suitable comment for the quick fix. And > I suppose effective_io_concurrency is a reasonable default. > > It's pretty hard to think of a good way to get your hands on

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-03-29 Thread Andres Freund
On 2019-03-29 16:20:54 +, Simon Riggs wrote: > On Fri, 29 Mar 2019 at 16:12, Andres Freund wrote: > > > > On 2019-03-29 15:58:14 +, Simon Riggs wrote: > > > On Fri, 29 Mar 2019 at 15:29, Andres Freund wrote: > > > > That's far from a trivial feature imo. It seems quite possible that >

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-03-29 Thread Peter Geoghegan
On Fri, Mar 29, 2019 at 9:12 AM Andres Freund wrote: > But even so, you can't have unlogged changes that you then rely on. Even > if there's no torn page issue. Currently BTP_HAS_GARBAGE and > ItemIdMarkDead() are treated as hints - if we want to guarantee all > these are accurate, I don't quite

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-03-29 Thread Simon Riggs
On Fri, 29 Mar 2019 at 16:12, Andres Freund wrote: > On 2019-03-29 15:58:14 +, Simon Riggs wrote: > > On Fri, 29 Mar 2019 at 15:29, Andres Freund wrote: > > > That's far from a trivial feature imo. It seems quite possible that > we'd > > > end up with increased overhead, because the

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-03-29 Thread Andres Freund
Hi, On 2019-03-29 15:58:14 +, Simon Riggs wrote: > On Fri, 29 Mar 2019 at 15:29, Andres Freund wrote: > > That's far from a trivial feature imo. It seems quite possible that we'd > > end up with increased overhead, because the current logic can get away > > with only doing hint bit style

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-03-29 Thread Simon Riggs
On Fri, 29 Mar 2019 at 15:29, Andres Freund wrote: > On 2019-03-29 09:37:11 +, Simon Riggs wrote: > > > While trying to understand this, I see there is an even better way to > > optimize this. Since we are removing dead index tuples, we could alter > the > > killed index tuple interface

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-03-29 Thread Andres Freund
Hi, On 2019-03-29 09:37:11 +, Simon Riggs wrote: > This commit message was quite confusing. It took me a while to realize this > relates to btree index deletes and that what you mean is that we are > calculcating the latestRemovedXid for index entries. That is related to but > not same thing

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-03-29 Thread Simon Riggs
On Wed, 27 Mar 2019 at 00:06, Andres Freund wrote: > Compute XID horizon for page level index vacuum on primary. > > Previously the xid horizon was only computed during WAL replay. This commit message was quite confusing. It took me a while to realize this relates to btree index deletes and