Re: Tracking last scan time

2022-11-08 Thread Dave Page
On Tue, 8 Nov 2022 at 04:10, Michael Paquier wrote: > On Mon, Nov 07, 2022 at 04:54:07PM +0900, Michael Paquier wrote: > > FWIW, all the other areas of pgstatfuncs.c manipulate timestamptz > > fields with a style like the attached. That's a nit, still per the > > role of consistency with the sur

Re: Tracking last scan time

2022-11-07 Thread Michael Paquier
On Mon, Nov 07, 2022 at 04:54:07PM +0900, Michael Paquier wrote: > FWIW, all the other areas of pgstatfuncs.c manipulate timestamptz > fields with a style like the attached. That's a nit, still per the > role of consistency with the surroundings.. > > Anyway, it seems to me that a regression test

Re: Tracking last scan time

2022-11-06 Thread Michael Paquier
On Thu, Nov 03, 2022 at 04:44:16PM -0400, Dave Page wrote: > Here's a patch to fix this issue. Many thanks to Peter Eisentraut who > figured it out in a few minutes after I spent far too long looking down > rabbit holes in entirely the wrong place. FWIW, all the other areas of pgstatfuncs.c manipu

Re: Tracking last scan time

2022-11-03 Thread Dave Page
On Mon, 31 Oct 2022 at 07:36, Dave Page wrote: > FYI, this is not intentional, and I do plan to look into it, however I've > been somewhat busy with pgconfeu, and am travelling for the rest of this > week as well. > Here's a patch to fix this issue. Many thanks to Peter Eisentraut who figured it

Re: Tracking last scan time

2022-10-31 Thread Dave Page
FYI, this is not intentional, and I do plan to look into it, however I've been somewhat busy with pgconfeu, and am travelling for the rest of this week as well. On Sun, 23 Oct 2022 at 21:09, Robert Treat wrote: > On Fri, Oct 14, 2022 at 2:55 PM Dave Page wrote: > > On Fri, 14 Oct 2022 at 19:16,

Re: Tracking last scan time

2022-10-23 Thread Robert Treat
On Fri, Oct 14, 2022 at 2:55 PM Dave Page wrote: > On Fri, 14 Oct 2022 at 19:16, Andres Freund wrote: >> On 2022-10-13 14:38:06 +0100, Dave Page wrote: >> > Thanks for that. It looks good to me, bar one comment (repeated 3 times in >> > the sql and expected files): >> > >> > fetch timestamps from

Re: Tracking last scan time

2022-10-14 Thread Dave Page
On Fri, 14 Oct 2022 at 19:16, Andres Freund wrote: > Hi, > > On 2022-10-13 14:38:06 +0100, Dave Page wrote: > > Thanks for that. It looks good to me, bar one comment (repeated 3 times > in > > the sql and expected files): > > > > fetch timestamps from before the next test > > > > "from " should b

Re: Tracking last scan time

2022-10-14 Thread Andres Freund
Hi, On 2022-10-13 14:38:06 +0100, Dave Page wrote: > Thanks for that. It looks good to me, bar one comment (repeated 3 times in > the sql and expected files): > > fetch timestamps from before the next test > > "from " should be removed. I was trying to say something with that from, but clearly

Re: Tracking last scan time

2022-10-13 Thread Dave Page
Hi On Wed, 12 Oct 2022 at 23:52, Andres Freund wrote: > Hi, > > On 2022-10-12 12:50:31 -0700, Andres Freund wrote: > > I think this should have at a basic test in > src/test/regress/sql/stats.sql. If > > I can write one in a few minutes I'll go for that, otherwise will reply > > detailing diffic

Re: Tracking last scan time

2022-10-12 Thread Andres Freund
Hi, On 2022-10-12 12:50:31 -0700, Andres Freund wrote: > I think this should have at a basic test in src/test/regress/sql/stats.sql. If > I can write one in a few minutes I'll go for that, otherwise will reply > detailing difficulties. Took a bit longer (+lunch). Attached. In the attached 0001,

Re: Tracking last scan time

2022-10-12 Thread Andres Freund
Hi, On 2022-10-12 15:40:21 +0900, Michael Paquier wrote: > On Mon, Oct 03, 2022 at 12:55:40PM +0100, Dave Page wrote: > > Thanks. It's just the changes in xact.c, so it doesn't seem like it would > > cause you any more work either way, in which case, I'll leave it to you :-) > > Okay, I have just

Re: Tracking last scan time

2022-10-12 Thread Michael Paquier
On Wed, Oct 12, 2022 at 09:09:46AM +0100, Dave Page wrote: > On Wed, 12 Oct 2022 at 07:40, Michael Paquier wrote: >> Okay, I have just moved the patch to the next CF then, still marked as >> ready for committer. Are you planning to look at that? > > Thanks. Was the question directed at me or Andr

Re: Tracking last scan time

2022-10-12 Thread Dave Page
On Wed, 12 Oct 2022 at 07:40, Michael Paquier wrote: > On Mon, Oct 03, 2022 at 12:55:40PM +0100, Dave Page wrote: > > Thanks. It's just the changes in xact.c, so it doesn't seem like it would > > cause you any more work either way, in which case, I'll leave it to you > :-) > > Okay, I have just m

Re: Tracking last scan time

2022-10-11 Thread Michael Paquier
On Mon, Oct 03, 2022 at 12:55:40PM +0100, Dave Page wrote: > Thanks. It's just the changes in xact.c, so it doesn't seem like it would > cause you any more work either way, in which case, I'll leave it to you :-) Okay, I have just moved the patch to the next CF then, still marked as ready for comm

Re: Tracking last scan time

2022-10-03 Thread Dave Page
Hi On Fri, 30 Sept 2022 at 18:58, Andres Freund wrote: > Hi, > > On 2022-09-30 17:58:31 +0200, Vik Fearing wrote: > > On 9/7/22 12:03, Dave Page wrote: > > > Here's a v4 patch. This reverts to using > > > GetCurrentTransactionStopTimestamp() for the last_scan times, and will > > > set xactStopTi

Re: Tracking last scan time

2022-09-30 Thread Andres Freund
Hi, On 2022-09-30 17:58:31 +0200, Vik Fearing wrote: > On 9/7/22 12:03, Dave Page wrote: > > Here's a v4 patch. This reverts to using > > GetCurrentTransactionStopTimestamp() for the last_scan times, and will > > set xactStopTimestamp the first time GetCurrentTransactionStopTimestamp() > > is call

Re: Tracking last scan time

2022-09-30 Thread Vik Fearing
On 9/7/22 12:03, Dave Page wrote: Here's a v4 patch. This reverts to using GetCurrentTransactionStopTimestamp() for the last_scan times, and will set xactStopTimestamp the first time GetCurrentTransactionStopTimestamp() is called, thus avoiding multiple gettimeofday() calls. SetCurrentTransaction

Re: Tracking last scan time

2022-09-07 Thread Dave Page
Hi On Tue, 6 Sept 2022 at 16:53, Andres Freund wrote: > Hi, > > On 2022-09-06 14:15:56 +0100, Dave Page wrote: > > Vik and I looked at this a little, and found that we actually don't have > > generally have GetCurrentTransactionStopTimestamp() at this point - a > > simple 'select * from pg_class

Re: Tracking last scan time

2022-09-07 Thread Kyotaro Horiguchi
At Tue, 6 Sep 2022 08:53:25 -0700, Andres Freund wrote in > Hi, > > On 2022-09-06 14:15:56 +0100, Dave Page wrote: > > Vik and I looked at this a little, and found that we actually don't have > > generally have GetCurrentTransactionStopTimestamp() at this point - a > > simple 'select * from pg_c

Re: Tracking last scan time

2022-09-06 Thread Andres Freund
Hi, On 2022-09-06 14:15:56 +0100, Dave Page wrote: > Vik and I looked at this a little, and found that we actually don't have > generally have GetCurrentTransactionStopTimestamp() at this point - a > simple 'select * from pg_class' will result in 9 passes of this code, none > of which have xactSto

Re: Tracking last scan time

2022-09-06 Thread Dave Page
Hi On Thu, 1 Sept 2022 at 19:35, Andres Freund wrote: > Hi, > > On 2022-09-01 14:18:42 +0200, Matthias van de Meent wrote: > > On Wed, 31 Aug 2022 at 20:56, Andres Freund wrote: > > > But given this is done when stats are flushed, which only happens > after the > > > transaction ended, we can j

Re: Tracking last scan time

2022-09-01 Thread Andres Freund
Hi, On 2022-09-01 14:18:42 +0200, Matthias van de Meent wrote: > On Wed, 31 Aug 2022 at 20:56, Andres Freund wrote: > > But given this is done when stats are flushed, which only happens after the > > transaction ended, we can just use GetCurrentTransactionStopTimestamp() - if > > we got to flushi

Re: Tracking last scan time

2022-09-01 Thread Matthias van de Meent
On Wed, 31 Aug 2022 at 20:56, Andres Freund wrote: > > Hi, > > On 2022-08-31 19:52:49 +0200, Matthias van de Meent wrote: > > As for having a lower granularity and preventing the > > one-syscall-per-Relation issue, can't we reuse the query_start or > > state_change timestamps that appear in pg_sta

Re: Tracking last scan time

2022-09-01 Thread Dave Page
On Thu, 1 Sept 2022 at 13:04, Bruce Momjian wrote: > On Thu, Sep 1, 2022 at 09:46:59AM +0100, Dave Page wrote: > > On Wed, 31 Aug 2022 at 17:13, Bruce Momjian wrote: > > Wow. I was just thinking you need second-level accuracy, which must > be > > cheap somewhere. > > > > > > Second-lev

Re: Tracking last scan time

2022-09-01 Thread Bruce Momjian
On Thu, Sep 1, 2022 at 09:46:59AM +0100, Dave Page wrote: > On Wed, 31 Aug 2022 at 17:13, Bruce Momjian wrote: > Wow.  I was just thinking you need second-level accuracy, which must be > cheap somewhere. > > > Second-level accuracy would indeed be fine for this. Frankly, for my use case

Re: Tracking last scan time

2022-09-01 Thread Dave Page
On Wed, 31 Aug 2022 at 17:13, Bruce Momjian wrote: > On Wed, Aug 31, 2022 at 05:02:33PM +0100, Dave Page wrote: > > > > > > On Tue, 30 Aug 2022 at 19:46, Bruce Momjian wrote: > > > > On Fri, Aug 26, 2022 at 02:05:36PM +0100, Dave Page wrote: > > > On Thu, 25 Aug 2022 at 01:44, David Rowl

Re: Tracking last scan time

2022-08-31 Thread Bruce Momjian
On Wed, Aug 31, 2022 at 11:56:29AM -0700, Andres Freund wrote: > Hi, > > On 2022-08-31 19:52:49 +0200, Matthias van de Meent wrote: > > As for having a lower granularity and preventing the > > one-syscall-per-Relation issue, can't we reuse the query_start or > > state_change timestamps that appear

Re: Tracking last scan time

2022-08-31 Thread Andres Freund
Hi, On 2022-08-31 19:52:49 +0200, Matthias van de Meent wrote: > As for having a lower granularity and preventing the > one-syscall-per-Relation issue, can't we reuse the query_start or > state_change timestamps that appear in pg_stat_activity (potentially > updated immediately before this stat fl

Re: Tracking last scan time

2022-08-31 Thread Bruce Momjian
On Wed, Aug 31, 2022 at 07:52:49PM +0200, Matthias van de Meent wrote: > As for having a lower granularity and preventing the > one-syscall-per-Relation issue, can't we reuse the query_start or > state_change timestamps that appear in pg_stat_activity (potentially Yeah, query start should be fine,

Re: Tracking last scan time

2022-08-31 Thread Matthias van de Meent
On Wed, 31 Aug 2022 at 18:21, Andres Freund wrote: > > Hi, > > On 2022-08-23 10:55:09 +0100, Dave Page wrote: > > Often it is beneficial to review one's schema with a view to removing > > indexes (and sometimes tables) that are no longer required. It's very > > difficult to understand when that is

Re: Tracking last scan time

2022-08-31 Thread Andres Freund
Hi, On 2022-08-23 10:55:09 +0100, Dave Page wrote: > Often it is beneficial to review one's schema with a view to removing > indexes (and sometimes tables) that are no longer required. It's very > difficult to understand when that is the case by looking at the number of > scans of a relation as, f

Re: Tracking last scan time

2022-08-31 Thread Bruce Momjian
On Wed, Aug 31, 2022 at 05:02:33PM +0100, Dave Page wrote: > > > On Tue, 30 Aug 2022 at 19:46, Bruce Momjian wrote: > > On Fri, Aug 26, 2022 at 02:05:36PM +0100, Dave Page wrote: > > On Thu, 25 Aug 2022 at 01:44, David Rowley wrote: > >     I don't have a particular opinion about t

Re: Tracking last scan time

2022-08-31 Thread Dave Page
On Tue, 30 Aug 2022 at 19:46, Bruce Momjian wrote: > On Fri, Aug 26, 2022 at 02:05:36PM +0100, Dave Page wrote: > > On Thu, 25 Aug 2022 at 01:44, David Rowley wrote: > > I don't have a particular opinion about the patch, I'm just pointing > > out that there are other ways. Even just writ

Re: Tracking last scan time

2022-08-30 Thread Bruce Momjian
On Fri, Aug 26, 2022 at 02:05:36PM +0100, Dave Page wrote: > On Thu, 25 Aug 2022 at 01:44, David Rowley wrote: > I don't have a particular opinion about the patch, I'm just pointing > out that there are other ways. Even just writing down the numbers on a > post-it note and coming back

Re: Tracking last scan time

2022-08-26 Thread Dave Page
Hi On Thu, 25 Aug 2022 at 01:44, David Rowley wrote: > On Thu, 25 Aug 2022 at 03:03, Bruce Momjian wrote: > > > > On Wed, Aug 24, 2022 at 04:01:21PM +0100, Dave Page wrote: > > > On Wed, 24 Aug 2022 at 15:18, Bruce Momjian wrote: > > > Would it be simpler to allow the sequential and index

Re: Tracking last scan time

2022-08-24 Thread David Rowley
On Thu, 25 Aug 2022 at 03:03, Bruce Momjian wrote: > > On Wed, Aug 24, 2022 at 04:01:21PM +0100, Dave Page wrote: > > On Wed, 24 Aug 2022 at 15:18, Bruce Momjian wrote: > > Would it be simpler to allow the sequential and index scan columns to be > > cleared so you can look later to see if

Re: Tracking last scan time

2022-08-24 Thread Dave Page
On Wed, 24 Aug 2022 at 16:03, Bruce Momjian wrote: > On Wed, Aug 24, 2022 at 04:01:21PM +0100, Dave Page wrote: > > On Wed, 24 Aug 2022 at 15:18, Bruce Momjian wrote: > > > > On Tue, Aug 23, 2022 at 10:55:09AM +0100, Dave Page wrote: > > > Often it is beneficial to review one's schema wi

Re: Tracking last scan time

2022-08-24 Thread Bruce Momjian
On Wed, Aug 24, 2022 at 04:01:21PM +0100, Dave Page wrote: > On Wed, 24 Aug 2022 at 15:18, Bruce Momjian wrote: > > On Tue, Aug 23, 2022 at 10:55:09AM +0100, Dave Page wrote: > > Often it is beneficial to review one's schema with a view to removing > indexes > > (and sometimes tab

Re: Tracking last scan time

2022-08-24 Thread Dave Page
On Wed, 24 Aug 2022 at 15:18, Bruce Momjian wrote: > On Tue, Aug 23, 2022 at 10:55:09AM +0100, Dave Page wrote: > > Often it is beneficial to review one's schema with a view to removing > indexes > > (and sometimes tables) that are no longer required. It's very difficult > to > > understand when

Re: Tracking last scan time

2022-08-24 Thread Bruce Momjian
On Tue, Aug 23, 2022 at 10:55:09AM +0100, Dave Page wrote: > Often it is beneficial to review one's schema with a view to removing indexes > (and sometimes tables) that are no longer required. It's very difficult to > understand when that is the case by looking at the number of scans of a > relatio

Re: Tracking last scan time

2022-08-24 Thread Dave Page
Hi On Tue, 23 Aug 2022 at 13:07, Greg Stark wrote: > On Tue, 23 Aug 2022 at 11:00, Dave Page wrote: > > > > Often it is beneficial to review one's schema with a view to removing > indexes (and sometimes tables) that are no longer required. It's very > difficult to understand when that is the ca

Re: Tracking last scan time

2022-08-23 Thread Greg Stark
On Tue, 23 Aug 2022 at 11:00, Dave Page wrote: > > Often it is beneficial to review one's schema with a view to removing indexes > (and sometimes tables) that are no longer required. It's very difficult to > understand when that is the case by looking at the number of scans of a > relation as,

Tracking last scan time

2022-08-23 Thread Dave Page
Often it is beneficial to review one's schema with a view to removing indexes (and sometimes tables) that are no longer required. It's very difficult to understand when that is the case by looking at the number of scans of a relation as, for example, an index may be used infrequently but may be cri