Re: Proposal to add page headers to SLRU pages

On Mon, Jun 10, 2024 at 07:19:56AM +, Bertrand Drouvot wrote: > On Tue, Mar 19, 2024 at 06:48:33AM +, Li, Yong wrote: >> Unfortunately, the test requires a setup of two different versions of PG. I >> am not >> aware of an existing test infrastructure which can run automated tests using

Re: Proposal to add page headers to SLRU pages

Hi, On Tue, Mar 19, 2024 at 06:48:33AM +, Li, Yong wrote: > > Unfortunately, the test requires a setup of two different versions of PG. I > am not > aware of an existing test infrastructure which can run automated tests using > two > PGs. I did manually verify the output of pg_upgrade. I

Re: Proposal to add page headers to SLRU pages

>> - New comments have been added to pg_upgrade to mention the SLRU >> page header change as the reason for upgrading clog files. > > That seems reasonable, but were any alternatives discussed? Do we have > consensus that this is the right thing to do? In general, there are two approaches.

Re: Proposal to add page headers to SLRU pages

On Mon, 2024-03-11 at 10:01 +, Li, Yong wrote: > - The clog LSN group has been brought back. >   Now the page LSN on each clog page is used for honoring the write- > ahead rule >   and it is always the highest LSN of all the LSN groups on the page. >   The LSN groups are used by

Re: Proposal to add page headers to SLRU pages

> On Mar 9, 2024, at 05:22, Jeff Davis wrote: > > External Email > > On Wed, 2024-03-06 at 12:01 +, Li, Yong wrote: >> Rebase the patch against the latest HEAD. > > The upgrade logic could use more comments explaining what's going on > and why. As I understand it, it's a one-time

Re: Proposal to add page headers to SLRU pages

On Wed, 2024-03-06 at 12:01 +, Li, Yong wrote: > Rebase the patch against the latest HEAD. The upgrade logic could use more comments explaining what's going on and why. As I understand it, it's a one-time conversion that needs to happen between 16 and 17. Is that right? Was the way CLOG is

Re: Proposal to add page headers to SLRU pages

On Fri, 2024-03-08 at 12:39 -0800, Jeff Davis wrote: > Making a better secondary structure seems doable to me. Just to > brainstorm: We can also keep the group_lsns, and then just update both the CLOG page LSN and the group_lsn when setting the transaction status. The former would be used for all

Re: Proposal to add page headers to SLRU pages

On Fri, 2024-03-08 at 13:58 +0100, Alvaro Herrera wrote: > In the new code we effectively store only one LSN per page, which I > understand is strictly worse. To quote from the README: "Instead, we defer the setting of the hint bit if we have not yet flushed WAL as far as the LSN associated with

Re: Proposal to add page headers to SLRU pages

On 2024-Mar-08, Stephen Frost wrote: > Thanks for testing! That strikes me as perfectly reasonable and seems > unlikely that we'd get much benefit from parallelizing it, so I'd say it > makes sense to keep this code simple. Okay, agreed, that amount of time sounds reasonable to me too; but I

Re: Proposal to add page headers to SLRU pages

Greetings, * Li, Yong (y...@ebay.com) wrote: > > On Mar 7, 2024, at 03:09, Stephen Frost wrote: > > * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: > >> I suppose this is important to do if we ever want to move SLRUs into > >> shared buffers. However, I wonder about the extra time this adds

Re: Proposal to add page headers to SLRU pages

> On Mar 7, 2024, at 03:09, Stephen Frost wrote: > > External Email > > From: Stephen Frost > Subject: Re: Proposal to add page headers to SLRU pages > Date: March 7, 2024 at 03:09:59 GMT+8 > To: Alvaro Herrera > Cc: "Li, Yong" , Aleksander Alekseev >

Re: Proposal to add page headers to SLRU pages

Greetings, * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: > I suppose this is important to do if we ever want to move SLRUs into > shared buffers. However, I wonder about the extra time this adds to > pg_upgrade. Is this something we should be concerned about? Is there > any

Re: Proposal to add page headers to SLRU pages

Hello, I suppose this is important to do if we ever want to move SLRUs into shared buffers. However, I wonder about the extra time this adds to pg_upgrade. Is this something we should be concerned about? Is there any measurement/estimates to tell us how long this would be? Right now, if you

Re: Proposal to add page headers to SLRU pages

Rebase the patch against the latest HEAD. Regards, Yong slru_page_header_v5.patch Description: slru_page_header_v5.patch

Re: Proposal to add page headers to SLRU pages

Rebase the patch against the latest HEAD. Regards, Yong slru_page_header_v4.patch Description: slru_page_header_v4.patch

Re: Proposal to add page headers to SLRU pages

> On Jan 2, 2024, at 19:35, Aleksander Alekseev > wrote: > > Thanks for the updated patch. > > cfbot seems to have some complaints regarding compiler warnings and > also building the patch on Windows: > > http://cfbot.cputube.org/ Thanks for the information. Here is the updated patch.

Re: Proposal to add page headers to SLRU pages

Hi, > I have also added this thread to the current Commitfest and hope this patch > will be part of the 17 release. > > The commitfest link: > https://commitfest.postgresql.org/46/4709/ Thanks for the updated patch. cfbot seems to have some complaints regarding compiler warnings and also

Re: Proposal to add page headers to SLRU pages

> This work is being done in file.c – it seems to me the proper way to > proceed would be to continue writing on-disk upgrade logic here. > Besides that this looks good to me, would like to hear what others have to > say. Thank you, Rishu for taking time to review the code. I've updated the

Re: Proposal to add page headers to SLRU pages

On Thu, Dec 8, 2023 at 1:36 AM Li, Yong wrote: >Given so many different approaches were discussed, I have started a >wiki to record and collaborate all efforts towards SLRU >improvements. The wiki provides a concise overview of all the ideas >discussed and can serve as a portal for all

Re: Proposal to add page headers to SLRU pages

: Re: Proposal to add page headers to SLRU pages External Email On Thu, Dec 7, 2023 at 1:28 PM Debnath, Shawn wrote: > What is being proposed now is the simple and core functionality of introducing > page headers to SLRU pages while continuing to be in the SLRU cache. This > allows

Re: Proposal to add page headers to SLRU pages

On Thu, Dec 7, 2023 at 1:28 PM Debnath, Shawn wrote: > What is being proposed now is the simple and core functionality of introducing > page headers to SLRU pages while continuing to be in the SLRU cache. This > allows the whole project to be iterative and reviewers to better reason about > the

Re: Proposal to add page headers to SLRU pages

>> Sounds like a half-measure to me. If we really want to go down this >> rabbit hole IMO SLRU should be moved to shared buffers as proposed >> elsewhere [1]. > Thread that I cited stopped in 2018 for this exact reason. 5 years ago. Is > this argument still valid? Meanwhile checksums of buffer

Re: Proposal to add page headers to SLRU pages

07.12.2023, 19:17, "Aleksander Alekseev" :Hi, +1 to the idea to protect SLRUs from corruption. I'm slightly leaning towards the idea of separating checksums from data pages, but anyway this checksums are better than no checksums. On 7 Dec 2023, at 10:06, Li, Yong wrote: I am still

Re: Proposal to add page headers to SLRU pages

Hi, > +1 to the idea to protect SLRUs from corruption. I'm slightly leaning towards > the idea of separating checksums from data pages, but anyway this checksums > are better than no checksums. > > On 7 Dec 2023, at 10:06, Li, Yong wrote: > > I am still working on patching the pg_upgrade.

Re: Proposal to add page headers to SLRU pages

Hi Yong! +1 to the idea to protect SLRUs from corruption. I'm slightly leaning towards the idea of separating checksums from data pages, but anyway this checksums are better than no checksums. > On 7 Dec 2023, at 10:06, Li, Yong wrote: > > I am still working on patching the pg_upgrade. Just

Proposal to add page headers to SLRU pages

Hi all, PostgreSQL currently maintains several data structures in the SLRU cache. The current SLRU pages do not have any header, so it is impossible to checksum a page and verify its integrity. It is very difficult to debug issues caused by corrupted SLRU pages. Also, without a page header,