Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-31 Thread Decibel!
On Fri, Aug 31, 2007 at 03:11:29PM -0400, Tom Lane wrote: > Decibel! <[EMAIL PROTECTED]> writes: > > Even if we don't care about folks running on suspect hardware, having a > > CRC would make it far more reasonable to recommend full_page_writes=3Doff. > > This argument seems ridiculous. Finding o

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-31 Thread Tom Lane
Decibel! <[EMAIL PROTECTED]> writes: > Even if we don't care about folks running on suspect hardware, having a > CRC would make it far more reasonable to recommend full_page_writes=3Doff. This argument seems ridiculous. Finding out that you have corrupt data is no substitute for not having corrup

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-31 Thread Decibel!
On Fri, Aug 31, 2007 at 02:34:09PM +0200, Albe Laurenz wrote: > I have thought some more about it, and tend to agree now: > Checksums will only detect disk failure, and that's only > one of the many integrity problems that can happen. > And one that can be reduced to a reasonable degree with good >

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-31 Thread Albe Laurenz
Jan Wieck wrote: > Computing a checksum just before writing the block will NOT detect any > faulty memory or Postgres bug that corrupted the block. You will have a > perfectly fine checksum over the corrupted data. > > A checksum only detects corruptions that happen between write and read. > Mo

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-31 Thread Jan Wieck
On 8/28/2007 4:14 AM, Albe Laurenz wrote: Not all databases are on enterprise scale storage systems, and there's also the small possibility of PostgreSQL bugs that could be detected that way. Computing a checksum just before writing the block will NOT detect any faulty memory or Postgres bug t

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-29 Thread Florian Weimer
* Alban Hertroys: > If you have a "proper" production database server, your memory has > error checking, and your RAID controller has something of the kind > as well. To my knowledge, no readily available controller performs validation on reads (not even for RAID-1 or RAID-10, where it would be p

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-28 Thread Lincoln Yeoh
At 11:48 PM 8/27/2007, Trevor Talbot wrote: On 8/27/07, Jonah H. Harris <[EMAIL PROTECTED]> wrote: > On 8/27/07, Tom Lane <[EMAIL PROTECTED]> wrote: > > that and the lack of evidence that they'd actually gain anything > > I find it somewhat ironic that PostgreSQL strives to be fairly > non-corrup

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-28 Thread Albe Laurenz
Tom Lane wrote: Would it be an option to have a checksum somewhere in each data block that is verified upon read? > >>> That's been proposed before and rejected before. See the >>> archives ... > > I think > the prior discussions were around the same time WAL was initially put > in, a

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-27 Thread Decibel!
On Mon, Aug 27, 2007 at 12:08:17PM -0400, Jonah H. Harris wrote: > On 8/27/07, Tom Lane <[EMAIL PROTECTED]> wrote: > > Indeed. In fact, the most likely implementation of this (refuse to do > > anything with a page with a bad CRC) would be a net loss from that > > standpoint, because you couldn't g

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-27 Thread Jonah H. Harris
On 8/27/07, Tom Lane <[EMAIL PROTECTED]> wrote: > Indeed. In fact, the most likely implementation of this (refuse to do > anything with a page with a bad CRC) would be a net loss from that > standpoint, because you couldn't get *any* data out of a page, even if > only part of it had been zapped.

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-27 Thread Tom Lane
"Trevor Talbot" <[EMAIL PROTECTED]> writes: > On 8/27/07, Jonah H. Harris <[EMAIL PROTECTED]> wrote: >> I find it somewhat ironic that PostgreSQL strives to be fairly >> non-corruptable, yet has no way to detect a corrupted page. > But how does detecting a corrupted data page gain you any durabili

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-27 Thread Alban Hertroys
Jonah H. Harris wrote: > On 8/27/07, Tom Lane <[EMAIL PROTECTED]> wrote: >> that and the lack of evidence that they'd actually gain anything > > I find it somewhat ironic that PostgreSQL strives to be fairly > non-corruptable, yet has no way to detect a corrupted page. The only > reason for not h

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-27 Thread Trevor Talbot
On 8/27/07, Jonah H. Harris <[EMAIL PROTECTED]> wrote: > On 8/27/07, Tom Lane <[EMAIL PROTECTED]> wrote: > > that and the lack of evidence that they'd actually gain anything > > I find it somewhat ironic that PostgreSQL strives to be fairly > non-corruptable, yet has no way to detect a corrupted pa

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-27 Thread Jonah H. Harris
On 8/27/07, Tom Lane <[EMAIL PROTECTED]> wrote: > that and the lack of evidence that they'd actually gain anything I find it somewhat ironic that PostgreSQL strives to be fairly non-corruptable, yet has no way to detect a corrupted page. The only reason for not having CRCs is because it will slow

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-27 Thread Tom Lane
"Albe Laurenz" <[EMAIL PROTECTED]> writes: > Tom Lane wrote: >>> Would it be an option to have a checksum somewhere in each >>> data block that is verified upon read? >> That's been proposed before and rejected before. See the archives ... > I searched for "checksum" and couldn't find it. Could

Re: [HACKERS] [GENERAL] Undetected corruption of table files

2007-08-27 Thread Albe Laurenz
Tom Lane wrote: >> Would it be an option to have a checksum somewhere in each >> data block that is verified upon read? > > That's been proposed before and rejected before. See the archives ... I searched for "checksum" and couldn't find it. Could someone give me a pointer? I'm not talking about