Re: ReadRecentBuffer() doesn't scale well

2024-01-20 Thread vignesh C
On Fri, 30 Jun 2023 at 07:43, Thomas Munro wrote: > > On Fri, Jun 30, 2023 at 3:39 AM Andres Freund wrote: > > I am wondering if we don't want something more generic than stashing this in > > rd_amcache. Don't want to end up duplicating relevant code across the uses > > of > > rd_amcache in ever

Re: ReadRecentBuffer() doesn't scale well

2023-06-29 Thread Andres Freund
Hi, On 2023-06-30 14:13:11 +1200, Thomas Munro wrote: > On Fri, Jun 30, 2023 at 3:39 AM Andres Freund wrote: > > I am wondering if we don't want something more generic than stashing this in > > rd_amcache. Don't want to end up duplicating relevant code across the uses > > of > > rd_amcache in ev

Re: ReadRecentBuffer() doesn't scale well

2023-06-29 Thread Thomas Munro
On Fri, Jun 30, 2023 at 3:39 AM Andres Freund wrote: > I am wondering if we don't want something more generic than stashing this in > rd_amcache. Don't want to end up duplicating relevant code across the uses of > rd_amcache in every AM. I suppose we could try to track hot pages automatically. A

Re: ReadRecentBuffer() doesn't scale well

2023-06-29 Thread Andres Freund
Hi, On 2023-06-29 19:35:30 +1200, Thomas Munro wrote: > I (re)discovered why I used the lock-then-pin approach. In the > comments I mentioned InvalidBuffer(), but the main problem is in its > caller GetVictimBuffer() which has various sanity checks about > reference counts that can occasionally f

Re: ReadRecentBuffer() doesn't scale well

2023-06-29 Thread Thomas Munro
I (re)discovered why I used the lock-then-pin approach. In the comments I mentioned InvalidBuffer(), but the main problem is in its caller GetVictimBuffer() which has various sanity checks about reference counts that can occasionally fail if you have code randomly pinning any old buffer. New idea

Re: ReadRecentBuffer() doesn't scale well

2023-06-27 Thread Andres Freund
Hi, On 2023-06-27 01:10:25 -0700, Peter Geoghegan wrote: > On Mon, Jun 26, 2023 at 11:27 PM Andres Freund wrote: > > On 2023-06-26 21:53:12 -0700, Peter Geoghegan wrote: > > > It should be safe to allow searchers to see a version of the root page > > > that is out of date. The Lehman & Yao design

Re: ReadRecentBuffer() doesn't scale well

2023-06-27 Thread Andres Freund
Hi, On 2023-06-27 19:04:31 +0300, Ants Aasma wrote: > On Tue, 27 Jun 2023 at 18:40, Andres Freund wrote: > > On 2023-06-27 14:49:48 +0300, Ants Aasma wrote: > > > If you want to experiment, here is a rebased version of something I > > > hacked up a couple of years back on the way to Fosdem Pgday.

Re: ReadRecentBuffer() doesn't scale well

2023-06-27 Thread Ants Aasma
On Tue, 27 Jun 2023 at 18:40, Andres Freund wrote: > On 2023-06-27 14:49:48 +0300, Ants Aasma wrote: > > If you want to experiment, here is a rebased version of something I > > hacked up a couple of years back on the way to Fosdem Pgday. I didn't > > pursue it further because I didn't have a use c

Re: ReadRecentBuffer() doesn't scale well

2023-06-27 Thread Andres Freund
Hi, On 2023-06-27 14:49:48 +0300, Ants Aasma wrote: > If you want to experiment, here is a rebased version of something I > hacked up a couple of years back on the way to Fosdem Pgday. I didn't > pursue it further because I didn't have a use case where it showed a > significant difference. Thanks

Re: ReadRecentBuffer() doesn't scale well

2023-06-27 Thread Ants Aasma
On Tue, 27 Jun 2023 at 07:09, Andres Freund wrote: > On 2023-06-27 15:33:57 +1200, Thomas Munro wrote: > > On Tue, Jun 27, 2023 at 2:05 PM Andres Freund wrote: > > > Unfortunately it scaled way worse at first. This is not an inherent > > > issue, but > > > due to an implementation choice in Read

Re: ReadRecentBuffer() doesn't scale well

2023-06-27 Thread Peter Geoghegan
On Mon, Jun 26, 2023 at 11:27 PM Andres Freund wrote: > On 2023-06-26 21:53:12 -0700, Peter Geoghegan wrote: > > It should be safe to allow searchers to see a version of the root page > > that is out of date. The Lehman & Yao design is very permissive about > > these things. There aren't any speci

Re: ReadRecentBuffer() doesn't scale well

2023-06-26 Thread Andres Freund
Hi, On 2023-06-26 21:53:12 -0700, Peter Geoghegan wrote: > It should be safe to allow searchers to see a version of the root page > that is out of date. The Lehman & Yao design is very permissive about > these things. There aren't any special cases where the general rules > are weakened in some wa

Re: ReadRecentBuffer() doesn't scale well

2023-06-26 Thread Andres Freund
Hi, On 2023-06-27 16:40:08 +1200, Thomas Munro wrote: > On Tue, Jun 27, 2023 at 4:32 PM Peter Geoghegan wrote: > > On Mon, Jun 26, 2023 at 9:09 PM Andres Freund wrote: > > > I think we should be able to have a post-check that can figure out > > > if the copied root page is out of date after sear

Re: ReadRecentBuffer() doesn't scale well

2023-06-26 Thread Thomas Munro
On Tue, Jun 27, 2023 at 4:53 PM Peter Geoghegan wrote: > On Mon, Jun 26, 2023 at 9:40 PM Thomas Munro wrote: > > If the goal is to get rid of both pins and content locks, LSN isn't > > enough. A page might be evicted and replaced by another page that has > > the same LSN because they were modifi

Re: ReadRecentBuffer() doesn't scale well

2023-06-26 Thread Peter Geoghegan
On Mon, Jun 26, 2023 at 9:40 PM Thomas Munro wrote: > If the goal is to get rid of both pins and content locks, LSN isn't > enough. A page might be evicted and replaced by another page that has > the same LSN because they were modified by the same record. Maybe > that's vanishingly rare, but the

Re: ReadRecentBuffer() doesn't scale well

2023-06-26 Thread Thomas Munro
On Tue, Jun 27, 2023 at 4:32 PM Peter Geoghegan wrote: > On Mon, Jun 26, 2023 at 9:09 PM Andres Freund wrote: > > I think we should be able to have a post-check that can figure out > > if the copied root page is out of date after searching it, without needing > > the > > content lock. > > I'm gu

Re: ReadRecentBuffer() doesn't scale well

2023-06-26 Thread Peter Geoghegan
On Mon, Jun 26, 2023 at 9:09 PM Andres Freund wrote: > I think we should be able to have a post-check that can figure out > if the copied root page is out of date after searching it, without needing the > content lock. I'm guessing that you're thinking of doing something with the page LSN? -- P

Re: ReadRecentBuffer() doesn't scale well

2023-06-26 Thread Andres Freund
Hi, On 2023-06-27 15:33:57 +1200, Thomas Munro wrote: > On Tue, Jun 27, 2023 at 2:05 PM Andres Freund wrote: > > Unfortunately it scaled way worse at first. This is not an inherent issue, > > but > > due to an implementation choice in ReadRecentBuffer(). Whereas the normal > > BufferAlloc() pat

Re: ReadRecentBuffer() doesn't scale well

2023-06-26 Thread Peter Geoghegan
On Mon, Jun 26, 2023 at 8:34 PM Thomas Munro wrote: > Yeah. Aside from inherent nastiness of user-space spinlocks, this new > use case is also enormously more likely to contend and then get into > trouble by being preempted due to btree root pages being about the > hottest pages in the universe t

Re: ReadRecentBuffer() doesn't scale well

2023-06-26 Thread Thomas Munro
On Tue, Jun 27, 2023 at 2:05 PM Andres Freund wrote: > As mentioned nearby [1], Thomas brought up [2] the idea of using > ReadRecentBuffer() _bt_getroot(). I couldn't resist and prototyped it. Thanks! > Unfortunately it scaled way worse at first. This is not an inherent issue, but > due to an i

ReadRecentBuffer() doesn't scale well

2023-06-26 Thread Andres Freund
Hi, As mentioned nearby [1], Thomas brought up [2] the idea of using ReadRecentBuffer() _bt_getroot(). I couldn't resist and prototyped it. Unfortunately it scaled way worse at first. This is not an inherent issue, but due to an implementation choice in ReadRecentBuffer(). Whereas the normal Bu