Re: SLRUs in the main buffer pool - Page Header definitions

2023-09-27 Thread Robert Haas
On Fri, Sep 8, 2023 at 8:56 AM Stephen Frost wrote: > If we're going to effectively segregate the buffer pool into SLRU parts > vs. everything else and then use the existing strategies for SLRUs and > have that be different from what everything else is using ... then > that doesn't seem like it's

Re: SLRUs in the main buffer pool - Page Header definitions

2023-09-08 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Thu, Aug 24, 2023 at 3:28 PM Stephen Frost wrote: > > Agreed that we'd certainly want to make sure it's all correct and to do > > performance testing but in terms of how many buffers... isn't much of > > the point of this that we have

Re: SLRUs in the main buffer pool - Page Header definitions

2023-09-07 Thread Bagga, Rishu
Alekseev, Aleksander (aleksan...@timescale.com) wrote: > It looks like the patch in *this* thread was never registered on the > commitfest and/or tested by CF bot, unless I'm missing something. > Unfortunately it's a bit late to register it for the September CF > especially considering the fact

Re: SLRUs in the main buffer pool - Page Header definitions

2023-09-05 Thread Robert Haas
On Thu, Aug 24, 2023 at 3:28 PM Stephen Frost wrote: > Agreed that we'd certainly want to make sure it's all correct and to do > performance testing but in terms of how many buffers... isn't much of > the point of this that we have data in memory because it's being used > and if it's not then we

Re: SLRUs in the main buffer pool - Page Header definitions

2023-09-04 Thread Aleksander Alekseev
Hi, > Agreed that we'd certainly want to make sure it's all correct and to do > performance testing but in terms of how many buffers... isn't much of > the point of this that we have data in memory because it's being used > and if it's not then we evict it? That is, I wouldn't think we'd have >

Re: SLRUs in the main buffer pool - Page Header definitions

2023-08-24 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Fri, Aug 18, 2023 at 12:15 PM Nathan Bossart > wrote: > > I think I agree with Stephen. We routinely make changes that require > > updates to extensions, and I doubt anyone is terribly wild about > > maintaining two SLRU systems for

Re: SLRUs in the main buffer pool - Page Header definitions

2023-08-21 Thread Robert Haas
On Fri, Aug 18, 2023 at 12:15 PM Nathan Bossart wrote: > I think I agree with Stephen. We routinely make changes that require > updates to extensions, and I doubt anyone is terribly wild about > maintaining two SLRU systems for several years. Yeah, maintaining two systems doesn't sound like a

Re: SLRUs in the main buffer pool - Page Header definitions

2023-08-18 Thread Nathan Bossart
On Fri, Aug 18, 2023 at 08:12:41AM +, Bagga, Rishu wrote: > * Frost, Stephen (sfrowt(at)snowman(dot)net) wrote: >> Haven't really looked over the patches yet but I wanted to push back >> on this a bit- you're suggesting that we'd continue to maintain and >> update slru.c for the benefit of

Re: SLRUs in the main buffer pool - Page Header definitions

2023-07-17 Thread Stephen Frost
Greetings, [snipped quoted bits] Would really be helpful to keep who the author of each quoted snipper was when you quote them; dropping that makes it look like one person wrote all of them and that's confusing. * Bagga, Rishu (bagri...@amazon.com) wrote: > The third patch brings back the the

Re: SLRUs in the main buffer pool - Page Header definitions

2023-02-27 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Mon, Feb 27, 2023 at 8:56 AM Heikki Linnakangas wrote: > > I'm not sure if I like that or not. I think we should clean up and > > finish the other patches that this builds on first, and then decide if > > we want to use the standard

Re: SLRUs in the main buffer pool - Page Header definitions

2023-02-27 Thread Robert Haas
On Mon, Feb 27, 2023 at 8:56 AM Heikki Linnakangas wrote: > I'm not sure if I like that or not. I think we should clean up and > finish the other patches that this builds on first, and then decide if > we want to use the standard page header for the SLRUs or not. And if we > decide that we want

Re: SLRUs in the main buffer pool - Page Header definitions

2023-02-27 Thread Heikki Linnakangas
On 08/02/2023 22:26, Andres Freund wrote: On 2023-02-08 20:04:52 +, Bagga, Rishu wrote: To summarize, our underlying effort is to move the SLRUs to the buffer cache. We were working with Thomas Munro off a patch he introduced here [1]. Munro’s patch moves SLRUs to the buffer cache by

Re: SLRUs in the main buffer pool - Page Header definitions

2023-02-08 Thread Andres Freund
Hi, On 2023-02-08 20:04:52 +, Bagga, Rishu wrote: > To summarize, our underlying effort is to move the SLRUs to the buffer > cache. We were working with Thomas Munro off a patch he introduced here > [1]. Munro’s patch moves SLRUs to the buffer cache by introducing the > pseudo db id 9 to

Re: SLRUs in the main buffer pool - Page Header definitions

2023-02-07 Thread Andres Freund
Hi, On 2023-02-06 19:12:47 +, Bagga, Rishu wrote: > Rebased patch as per latest community changes since last email. This version doesn't actually build. https://cirrus-ci.com/task/4512310190931968 [19:43:20.131] FAILED: src/test/modules/test_slru/test_slru.so.p/test_slru.c.o

Re: SLRUs in the main buffer pool - Page Header definitions

2023-02-06 Thread Andres Freund
Hi, > From 098f37c0a17fc32a94bff43817414e01fcfb234f Mon Sep 17 00:00:00 2001 > From: Rishu Bagga > Date: Thu, 15 Sep 2022 00:55:25 + > Subject: [PATCH] slru to buffercache + page headers + upgrade > > --- > contrib/amcheck/verify_nbtree.c |2 +- > [...] > 65 files changed,

Re: SLRUs in the main buffer pool - Page Header definitions

2023-01-03 Thread vignesh C
On Fri, 16 Dec 2022 at 04:47, Bagga, Rishu wrote: > Rebased and updated a new patch addressing the critical section issue in > RecordNewMultliXact.In GetNewMultiXactId, we now make our ReadBuffer > calls before starting the critical section, but while holding the > MultiXactGenLock, so we always

Re: SLRUs in the main buffer pool - Page Header definitions

2022-11-03 Thread Ian Lawrence Barwick
2022年9月28日(水) 10:57 Bagga, Rishu : > > Hi, > > We have been working on adding page headers to the SLRU pages, as part of the > migration for SLRU to buffer cache. We’ve incorporated Thomas Munro’s patch > and Heikki’s Storage manager changes[1] and have a patch for early feedback. > > As part of

Re: SLRUs in the main buffer pool - Page Header definitions

2022-06-30 Thread Shawn Debnath
On Fri, Jun 24, 2022 at 03:45:34PM -0700, Andres Freund wrote: > Outside the database you'll know the path to the file, which will tell you > it's not another kind of relation. > > This really makes no sense to me. We don't have page flags indicating whether > a page is a heap, btree,

Re: SLRUs in the main buffer pool - Page Header definitions

2022-06-24 Thread Andres Freund
Hi, On 2022-06-24 22:19:33 +, Shawn Debnath wrote: > On Thu, Jun 23, 2022 at 06:06:48PM -0700, Andres Freund wrote: > > > > You are correct that we wouldn’t need to rely on the pd_flag bit to > > > determine page type for any access to a page where we come top down > > > following the

Re: SLRUs in the main buffer pool - Page Header definitions

2022-06-24 Thread Shawn Debnath
On Thu, Jun 23, 2022 at 06:06:48PM -0700, Andres Freund wrote: > > You are correct that we wouldn’t need to rely on the pd_flag bit to > > determine page type for any access to a page where we come top down > > following the hierarchy. However, for the purpose of debugging “from the > > bottom

Re: SLRUs in the main buffer pool - Page Header definitions

2022-06-23 Thread Andres Freund
Hi, On 2022-06-24 00:39:41 +, Bagga, Rishu wrote: > >When do we need to do so? We should never need to figure out whether an > >on-disk block is for an SLRU or something else, without also knowing >which > >relation / SLRU it is in. > > You are correct that we wouldn’t need to rely on the

Re: SLRUs in the main buffer pool - Page Header definitions

2022-06-23 Thread Bagga, Rishu
Hi Andres, >When do we need to do so? We should never need to figure out whether an >on-disk block is for an SLRU or something else, without also knowing >which >relation / SLRU it is in. You are correct that we wouldn’t need to rely on the pd_flag bit to determine page type for any access to

Re: SLRUs in the main buffer pool - Page Header definitions

2022-06-23 Thread Andres Freund
Hi, On 2022-06-23 20:25:21 +, Bagga, Rishu wrote: > >> 3. A flag to identify if the page is a relational or BufferedObject > >Why is this needed in the page header? > > Now that we are dealing with two different type of page headers, we need to > know how to interpret any given page. We need

Re: SLRUs in the main buffer pool - Page Header definitions

2022-06-23 Thread Robert Haas
On Wed, Jun 22, 2022 at 5:06 PM Bagga, Rishu wrote: > We are suggesting a minimal BufferedObject page header > to be the following, overlapping with the key fields near the beginning > of the regular PageHeaderData: > > typedef struct BufferedObjectPageHeaderData > { > PageXLogRecPtr pd_lsn;

Re: SLRUs in the main buffer pool - Page Header definitions

2022-06-23 Thread Bagga, Rishu
Hi Andres, Thanks for your response. To answer your questions: >> 3. A flag to identify if the page is a relational or BufferedObject >Why is this needed in the page header? Now that we are dealing with two different type of page headers, we need to know how to interpret any given page. We

Re: SLRUs in the main buffer pool - Page Header definitions

2022-06-22 Thread Andres Freund
Hi, On 2022-06-22 19:12:14 -0400, Tom Lane wrote: > "Bagga, Rishu" writes: > > The current SLRU pages do not have any header, so there is a need to > > create a new page header format for these. Our investigations revealed > > that we need to: > > > 1. track LSN to ensure durability and

Re: SLRUs in the main buffer pool - Page Header definitions

2022-06-22 Thread Tom Lane
"Bagga, Rishu" writes: > The current SLRU pages do not have any header, so there is a need to > create a new page header format for these. Our investigations revealed > that we need to: > 1. track LSN to ensure durability and consistency of all pages (for redo >    and full page write purposes)

Re: SLRUs in the main buffer pool - Page Header definitions

2022-06-22 Thread Andres Freund
Hi, On 2022-06-22 21:06:29 +, Bagga, Rishu wrote: > 3. A flag to identify if the page is a relational or BufferedObject Why is this needed in the page header? > Using the new BufferedObject page header will be space efficient but > introduces a significant change in the codebase to now

SLRUs in the main buffer pool - Page Header definitions

2022-06-22 Thread Bagga, Rishu
Hi all, PostgreSQL currently maintains several data structures in the SLRU cache. The SLRU cache has scaling and sizing challenges because of it’s simple implementation. The goal is to move these caches to the common buffer cache to benefit from the stronger capabilities of the common buffercache