Re: [HACKERS] Checksums by default?

On 01/21/2017 05:53 PM, Stephen Frost wrote: * Tom Lane (t...@sss.pgh.pa.us) wrote: Stephen Frost writes: * Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: The change of wal_level was supported by benchmark, I think it's reasonable to ask for this to be as well. No,

Re: [HACKERS] Checksums by default?

On 01/21/2017 05:51 PM, Stephen Frost wrote: * Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: On 21/01/17 17:31, Stephen Frost wrote: This is just changing the *default*, not requiring checksums to always be enabled. We do not hold the same standards for our defaults as we do for

Re: [HACKERS] Checksums by default?

On 01/21/2017 05:35 PM, Tom Lane wrote: Stephen Frost writes: * Tom Lane (t...@sss.pgh.pa.us) wrote: Have we seen *even one* report of checksums catching problems in auseful way? This isn't the right question. I disagree. If they aren't doing something useful for

Re: [HACKERS] Checksums by default?

On 1/21/17 10:02 AM, Tom Lane wrote: Magnus Hagander writes: Is it time to enable checksums by default, and give initdb a switch to turn it off instead? Have we seen *even one* report of checksums catching problems in a useful way? I've experienced multiple corruption

Re: [HACKERS] Checksums by default?

Tom, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Not at all; I just think that it's not clear that they are a net win > for the average user, and so I'm unconvinced that turning them on by > default is a good idea. I could be convinced otherwise by suitable > evidence. What I'm objecting to is

Re: [HACKERS] Checksums by default?

Thomas, * Thomas Munro (thomas.mu...@enterprisedb.com) wrote: > On Sun, Jan 22, 2017 at 7:37 AM, Stephen Frost wrote: > > Exactly, and that awareness will allow a user to prevent further data > > loss or corruption. Slow corruption over time is a very much known and > >

Re: [HACKERS] Checksums by default?

On Sun, Jan 22, 2017 at 7:37 AM, Stephen Frost wrote: > Exactly, and that awareness will allow a user to prevent further data > loss or corruption. Slow corruption over time is a very much known and > accepted real-world case that people do experience, as well as bit >

Re: [HACKERS] Checksums by default?

On Sat, Jan 21, 2017 at 10:16 PM, Michael Banck wrote: > On Sat, Jan 21, 2017 at 09:02:25PM +0200, Ants Aasma wrote: >> On Sat, Jan 21, 2017 at 6:41 PM, Andreas Karlsson wrote: >> > It might be worth looking into using the CRC CPU instruction to

Re: [HACKERS] Checksums by default?

-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Tom Lane points out: > Yeah, and there's a bunch of usability tooling that we don't have, > centered around "what do you do after you get a checksum error?". I've asked myself this as well, and came up with a proof of conecpt repair tool

Re: [HACKERS] Checksums by default?

On Sat, Jan 21, 2017 at 09:02:25PM +0200, Ants Aasma wrote: > On Sat, Jan 21, 2017 at 6:41 PM, Andreas Karlsson wrote: > > It might be worth looking into using the CRC CPU instruction to reduce this > > overhead, like we do for the WAL checksums. Since that is a different > >

Re: [HACKERS] Checksums by default?

-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 tl;dr +1 from me for changing the default, it is worth it. Tom Lane wrote: > Have we seen *even one* report of checksums catching > problems in a usefuld way? Sort of chicken-and-egg, as most places don't have it enabled. Which leads us to:

Re: [HACKERS] Checksums by default?

* Ants Aasma (ants.aa...@eesti.ee) wrote: > On Sat, Jan 21, 2017 at 7:39 PM, Petr Jelinek > wrote: > > So in summary "postgresql.conf options are easy to change" while "initdb > > options are hard to change", I can see this argument used both for > > enabling or

Re: [HACKERS] Checksums by default?

On Sat, Jan 21, 2017 at 7:39 PM, Petr Jelinek wrote: > So in summary "postgresql.conf options are easy to change" while "initdb > options are hard to change", I can see this argument used both for > enabling or disabling checksums by default. As I said I would be

Re: [HACKERS] Checksums by default?

On Sat, Jan 21, 2017 at 6:41 PM, Andreas Karlsson wrote: > On 01/21/2017 04:48 PM, Stephen Frost wrote: >> >> * Fujii Masao (masao.fu...@gmail.com) wrote: >>> >>> If the performance overhead by the checksums is really negligible, >>> we may be able to get rid of wal_log_hints

Re: [HACKERS] Checksums by default?

* Tom Lane (t...@sss.pgh.pa.us) wrote: > Andres Freund writes: > > Sure, it might be easy, but we don't have it. Personally I think > > checksums just aren't even ready for prime time. If we had: > > - ability to switch on/off at runtime (early patches for that have IIRC > >

Re: [HACKERS] Checksums by default?

* Andres Freund (and...@anarazel.de) wrote: > On 2017-01-21 13:03:52 -0500, Stephen Frost wrote: > > * Andres Freund (and...@anarazel.de) wrote: > > > On 2017-01-21 12:46:05 -0500, Stephen Frost wrote: > > > > Do you run with all defaults in those environments? > > > > > > Irrelevant - changing

Re: [HACKERS] Checksums by default?

* Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > Because I see having checksums as, frankly, something we always should > > have had (as most other databases do, for good reason...) and because > > they will hopefully prevent data loss. I'm willing to give

Re: [HACKERS] Checksums by default?

Andres Freund writes: > Sure, it might be easy, but we don't have it. Personally I think > checksums just aren't even ready for prime time. If we had: > - ability to switch on/off at runtime (early patches for that have IIRC > been posted) > - *builtin* tooling to check

Re: [HACKERS] Checksums by default?

On 2017-01-21 13:04:18 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2017-01-21 12:46:05 -0500, Stephen Frost wrote: > >> Do you run with all defaults in those environments? > > > Irrelevant - changing requires re-initdb'ing. That's unrealistic. > > If you can't turn

Re: [HACKERS] Checksums by default?

On 2017-01-21 13:03:52 -0500, Stephen Frost wrote: > * Andres Freund (and...@anarazel.de) wrote: > > On 2017-01-21 12:46:05 -0500, Stephen Frost wrote: > > > Do you run with all defaults in those environments? > > > > Irrelevant - changing requires re-initdb'ing. That's unrealistic. > > I

Re: [HACKERS] Checksums by default?

Stephen Frost writes: > Because I see having checksums as, frankly, something we always should > have had (as most other databases do, for good reason...) and because > they will hopefully prevent data loss. I'm willing to give us a fair > bit to minimize the risk of losing

Re: [HACKERS] Checksums by default?

* Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: > > Do you run with all defaults in those environments? > > For initdb? Mostly yes. Ok, fine, but you probably wouldn't if this change went in. For me, it's the other way- I have to go enable checksums at initdb time unless there's an excuse

Re: [HACKERS] Checksums by default?

Andres Freund writes: > On 2017-01-21 12:46:05 -0500, Stephen Frost wrote: >> Do you run with all defaults in those environments? > Irrelevant - changing requires re-initdb'ing. That's unrealistic. If you can't turn checksums *off* without re-initdb, that raises the stakes

Re: [HACKERS] Checksums by default?

On 2017-01-21 12:46:05 -0500, Stephen Frost wrote: > > I stand by the opinion that changing default which affect performance > > without any benchmark is bad idea. > > I'd be surprised if the performance impact has really changed all that > much since the code went in. Perhaps that's overly

Re: [HACKERS] Checksums by default?

On 21/01/17 18:46, Stephen Frost wrote: > * Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: >> As we don't know the performance impact is (there was no benchmark done >> on reasonably current code base) I really don't understand how you can >> judge if it's worth it or not. > > Because I see

Re: [HACKERS] Checksums by default?

* Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: > As we don't know the performance impact is (there was no benchmark done > on reasonably current code base) I really don't understand how you can > judge if it's worth it or not. Because I see having checksums as, frankly, something we always

Re: [HACKERS] Checksums by default?

On 21/01/17 18:15, Stephen Frost wrote: > * Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: >> On 21/01/17 17:51, Stephen Frost wrote: >>> I'm quite sure that the performance numbers for the CREATE TABLE + COPY >>> case with wal_level=minimal would have been *far* better than for >>> wal_level

Re: [HACKERS] Checksums by default?

Andres Freund writes: > What wouldn't hurt is enabling it by default in pg_regress on master for > a while. That seems like a good thing to do independent of flipping the > default. Yeah, I could get behind that. I'm not certain how much the regression tests really stress

Re: [HACKERS] Checksums by default?

* Andres Freund (and...@anarazel.de) wrote: > On 2017-01-21 12:09:53 -0500, Tom Lane wrote: > > Also, if we do decide to do that, there's the question of timing. > > As I mentioned, one of the chief risks I see is the possibility of > > false-positive checksum failures due to bugs; I think that

Re: [HACKERS] Checksums by default?

* Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > As for checksums, I do see value in them and I'm pretty sure that the > > author of that particular feature did as well, or we wouldn't even have > > it as an option. You seem to be of the opinion that we

Re: [HACKERS] Checksums by default?

* Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: > On 21/01/17 17:51, Stephen Frost wrote: > > I'm quite sure that the performance numbers for the CREATE TABLE + COPY > > case with wal_level=minimal would have been *far* better than for > > wal_level > minimal. > > Which is random usecase

Re: [HACKERS] Checksums by default?

On 2017-01-21 12:09:53 -0500, Tom Lane wrote: > Also, if we do decide to do that, there's the question of timing. > As I mentioned, one of the chief risks I see is the possibility of > false-positive checksum failures due to bugs; I think that code has seen > sufficiently little field use that we

Re: [HACKERS] Checksums by default?

Stephen Frost writes: > As for checksums, I do see value in them and I'm pretty sure that the > author of that particular feature did as well, or we wouldn't even have > it as an option. You seem to be of the opinion that we might as well > just rip all of that code and work

Re: [HACKERS] Checksums by default?

On 21/01/17 17:51, Stephen Frost wrote: > * Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: >> On 21/01/17 17:31, Stephen Frost wrote: >>> This is just changing the *default*, not requiring checksums to always >>> be enabled. We do not hold the same standards for our defaults as we do >>> for

Re: [HACKERS] Checksums by default?

On 2017-01-22 00:41:55 +0900, Fujii Masao wrote: > On Sun, Jan 22, 2017 at 12:18 AM, Petr Jelinek > wrote: > > On 21/01/17 11:39, Magnus Hagander wrote: > >> Is it time to enable checksums by default, and give initdb a switch to > >> turn it off instead? > > > > I'd

Re: [HACKERS] Checksums by default?

On 2017-01-21 11:39:18 +0100, Magnus Hagander wrote: > Is it time to enable checksums by default, and give initdb a switch to turn > it off instead? -1 - the WAL overhead is quite massive, and in contrast to the other GUCs recently changed you can't just switch this around. Andres -- Sent via

Re: [HACKERS] Checksums by default?

* Andreas Karlsson (andr...@proxel.se) wrote: > On 01/21/2017 04:48 PM, Stephen Frost wrote: > >* Fujii Masao (masao.fu...@gmail.com) wrote: > >>If the performance overhead by the checksums is really negligible, > >>we may be able to get rid of wal_log_hints parameter, as well. > > > >Prior

Re: [HACKERS] Checksums by default?

* Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > * Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: > >> The change of wal_level was supported by benchmark, I think it's > >> reasonable to ask for this to be as well. > > > No, it wasn't, it was that people

Re: [HACKERS] Checksums by default?

* Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: > On 21/01/17 17:31, Stephen Frost wrote: > > This is just changing the *default*, not requiring checksums to always > > be enabled. We do not hold the same standards for our defaults as we do > > for always-enabled code, for clear reasons- not

Re: [HACKERS] Checksums by default?

Stephen Frost writes: > * Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: >> The change of wal_level was supported by benchmark, I think it's >> reasonable to ask for this to be as well. > No, it wasn't, it was that people felt the cases where changing > wal_level would

Re: [HACKERS] Checksums by default?

On 01/21/2017 04:48 PM, Stephen Frost wrote: * Fujii Masao (masao.fu...@gmail.com) wrote: If the performance overhead by the checksums is really negligible, we may be able to get rid of wal_log_hints parameter, as well. Prior benchmarks showed it to be on the order of a few percent, as I

Re: [HACKERS] Checksums by default?

On 21/01/17 17:31, Stephen Frost wrote: > Petr, > > * Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: >> On 21/01/17 16:40, Stephen Frost wrote: >>> * Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: On 21/01/17 11:39, Magnus Hagander wrote: > Is it time to enable checksums by

Re: [HACKERS] Checksums by default?

Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> Have we seen *even one* report of checksums catching problems in a useful >> way? > This isn't the right question. I disagree. If they aren't doing something useful for people who have turned them on, what's

Re: [HACKERS] Checksums by default?

Petr, * Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: > On 21/01/17 16:40, Stephen Frost wrote: > > * Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: > >> On 21/01/17 11:39, Magnus Hagander wrote: > >>> Is it time to enable checksums by default, and give initdb a switch to > >>> turn it

Re: [HACKERS] Checksums by default?

On 21/01/17 16:40, Stephen Frost wrote: > Petr, > > * Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: >> On 21/01/17 11:39, Magnus Hagander wrote: >>> Is it time to enable checksums by default, and give initdb a switch to >>> turn it off instead? >> >> I'd like to see benchmark first, both in

Re: [HACKERS] Checksums by default?

* Tom Lane (t...@sss.pgh.pa.us) wrote: > Magnus Hagander writes: > > Is it time to enable checksums by default, and give initdb a switch to turn > > it off instead? > > Have we seen *even one* report of checksums catching problems in a useful > way? This isn't the right

Re: [HACKERS] Checksums by default?

Magnus Hagander writes: > Is it time to enable checksums by default, and give initdb a switch to turn > it off instead? Have we seen *even one* report of checksums catching problems in a useful way? I think this will be making the average user pay X% for nothing.

Re: [HACKERS] Checksums by default?

* Fujii Masao (masao.fu...@gmail.com) wrote: > On Sun, Jan 22, 2017 at 12:18 AM, Petr Jelinek > wrote: > > On 21/01/17 11:39, Magnus Hagander wrote: > >> Is it time to enable checksums by default, and give initdb a switch to > >> turn it off instead? > > > > I'd like

Re: [HACKERS] Checksums by default?

On Sun, Jan 22, 2017 at 12:18 AM, Petr Jelinek wrote: > On 21/01/17 11:39, Magnus Hagander wrote: >> Is it time to enable checksums by default, and give initdb a switch to >> turn it off instead? > > I'd like to see benchmark first, both in terms of CPU and in terms

Re: [HACKERS] Checksums by default?

Petr, * Petr Jelinek (petr.jeli...@2ndquadrant.com) wrote: > On 21/01/17 11:39, Magnus Hagander wrote: > > Is it time to enable checksums by default, and give initdb a switch to > > turn it off instead? > > I'd like to see benchmark first, both in terms of CPU and in terms of > produced WAL

Re: [HACKERS] Checksums by default?

On 21/01/17 11:39, Magnus Hagander wrote: > Is it time to enable checksums by default, and give initdb a switch to > turn it off instead? I'd like to see benchmark first, both in terms of CPU and in terms of produced WAL (=network traffic) given that it turns on logging of hint bits. -- Petr

Re: [HACKERS] Checksums by default?

Magnus, * Magnus Hagander (mag...@hagander.net) wrote: > On Sat, Jan 21, 2017 at 3:05 PM, Michael Paquier > wrote: > > > On Sat, Jan 21, 2017 at 7:39 PM, Magnus Hagander > > wrote: > > > Is it time to enable checksums by default, and give initdb

Re: [HACKERS] Checksums by default?

* Michael Paquier (michael.paqu...@gmail.com) wrote: > On Sat, Jan 21, 2017 at 7:39 PM, Magnus Hagander wrote: > > Is it time to enable checksums by default, and give initdb a switch to turn > > it off instead? > > > > I keep running into situations where people haven't

Re: [HACKERS] Checksums by default?

On Sat, Jan 21, 2017 at 3:05 PM, Michael Paquier wrote: > On Sat, Jan 21, 2017 at 7:39 PM, Magnus Hagander > wrote: > > Is it time to enable checksums by default, and give initdb a switch to > turn > > it off instead? > > > > I keep running into

Re: [HACKERS] Checksums by default?

On Sat, Jan 21, 2017 at 7:39 PM, Magnus Hagander wrote: > Is it time to enable checksums by default, and give initdb a switch to turn > it off instead? > > I keep running into situations where people haven't enabled it, because (a) > they didn't know about it, or (b) their

Re: [HACKERS] Checksums by default?

* Magnus Hagander (mag...@hagander.net) wrote: > Is it time to enable checksums by default, and give initdb a switch to turn > it off instead? Yes, please. We've already agreed to make changes to have a better user experience and ask those who really care about certain performance aspects to

[HACKERS] Checksums by default?

Is it time to enable checksums by default, and give initdb a switch to turn it off instead? I keep running into situations where people haven't enabled it, because (a) they didn't know about it, or (b) their packaging system ran initdb for them so they didn't even know they could. And of course

<    1   2