Re: [HACKERS] Online enabling of page level checksums

2017-01-24 Thread David Steele
On 1/23/17 9:45 PM, Jim Nasby wrote: > To put it another way, I think it's entirely reasonable *from a > technical standpoint* to enable by default in 10, with only the ability > to dynamically disable. Given the concerns that keep popping up about > dynamically enabling, I'm not at all sure that

Re: [HACKERS] Online enabling of page level checksums

2017-01-23 Thread Jim Nasby
On 1/23/17 12:03 PM, Greg Stark wrote: On Jan 22, 2017 11:13 AM, "Magnus Hagander" > wrote: Yes, this means the entire db will end up in the transaction log since everything is rewritten. That's not great, but for a lot of people

Re: [HACKERS] Online enabling of page level checksums

2017-01-23 Thread Tom Lane
Jim Nasby writes: > For a first pass, I think it's acceptable for autovac and vac to notice > if a relation needs checksums computed and ignore the VM in that case > (make sure it's ignoring all frozen bits too). > Likewise, for right now I think it's OK to force

Re: [HACKERS] Online enabling of page level checksums

2017-01-23 Thread Jim Nasby
On 1/23/17 1:11 PM, David Christensen wrote: I’m not sure that this will work as-is, unless we’re forcing VACUUM to ignore the visibility map. I had originally considered having this sit on top of VACUUm though, we just need a way to iterate over all relations and read every page. Another

Re: [HACKERS] Online enabling of page level checksums

2017-01-23 Thread David Christensen
> On Jan 23, 2017, at 10:59 AM, David Christensen wrote: > >> >> On Jan 23, 2017, at 10:53 AM, Simon Riggs wrote: >> >> On 23 January 2017 at 16:32, David Christensen wrote: >> >>> ** Handling checksums on a standby: >>> >>>

Re: [HACKERS] Online enabling of page level checksums

2017-01-23 Thread Greg Stark
On Jan 22, 2017 11:13 AM, "Magnus Hagander" wrote: Yes, this means the entire db will end up in the transaction log since everything is rewritten. That's not great, but for a lot of people that will be a trade they're willing to make since it's a one-time thing. Yes, this

Re: [HACKERS] Online enabling of page level checksums

2017-01-23 Thread David Christensen
> On Jan 23, 2017, at 10:53 AM, Simon Riggs wrote: > > On 23 January 2017 at 16:32, David Christensen wrote: > >> ** Handling checksums on a standby: >> >> How to handle checksums on a standby is a bit trickier since checksums are >> inherently a

Re: [HACKERS] Online enabling of page level checksums

2017-01-23 Thread Simon Riggs
On 23 January 2017 at 16:32, David Christensen wrote: > ** Handling checksums on a standby: > > How to handle checksums on a standby is a bit trickier since checksums are > inherently a local cluster state and not WAL logged but we are storing state > in the system tables

Re: [HACKERS] Online enabling of page level checksums

2017-01-23 Thread David Christensen
So as mentioned on IRC, I have a patch that I am working to rebase on HEAD with the following design. It is very similar to what you have proposed, so maybe we can use my development notes as a jumping-off point for discussion/refinement. * Incremental Checksums PostgreSQL users should have

Re: [HACKERS] Online enabling of page level checksums

2017-01-22 Thread Jim Nasby
On 1/22/17 5:13 AM, Magnus Hagander wrote: If the system is interrupted before the background worker is done, it starts over from the beginning. Previously touched blocks will be read and verified, but not written (because their checksum is already correct). This will take time, but not

[HACKERS] Online enabling of page level checksums

2017-01-22 Thread Magnus Hagander
So, that post I made about checksums certainly spurred a lot of discussion :) One summary is that life would be a lot easier if we could turn checksums on (and off) without re-initdbing. I'm breaking out this question into this thread to talk about it separately. I've been toying with a branch