> Instead of a partial row CRC, we could just as well use some other bit
> of identifying information, say the row OID. Given a block CRC on the
> heap page, we'll be pretty confident already that the heap page is OK,
> we just need to guard against the possibility that it's older than the
> inde
Andreas SB Zeugswetter wrote:
> Tom Lane wrote:
> > Instead of a partial row CRC, we could just as well use some other
> > bit of identifying information, say the row OID. ... Checking that
> > there is a valid tuple at the slot indicated by the index item,
> > and that it has the right OID, shoul
> Instead of a partial row CRC, we could just as well use some other bit
> of identifying information, say the row OID. Given a block CRC on the
> heap page, we'll be pretty confident already that the heap page is OK,
> we just need to guard against the possibility that it's older than the
> ind
[EMAIL PROTECTED] (Nathan Myers) writes:
>> Instead of a partial row CRC, we could just as well use some other bit
>> of identifying information, say the row OID. ...
> Good. But, wouldn't the TID be more specific?
Uh, the TID *is* the pointer from index to heap. There's no redundancy
that w
On Sunday 14 January 2001 04:49, Tom Lane wrote:
> A row-level CRC might be useful for this, but it would have to be on
> the data only (not the tuple commit-status bits). It'd be totally
> impractical with a block CRC, I think. To do it with a block CRC, every
> time you changed *anything* in
On Sat, Jan 13, 2001 at 12:49:34PM -0500, Tom Lane wrote:
> [EMAIL PROTECTED] (Nathan Myers) writes:
> > ... for systems that cannot provide strict write ordering (e.g.,
> > most PCs) it would be helpful to be able to detect that the database
> > has become corrupted. In Vadim's example above,
[EMAIL PROTECTED] (Nathan Myers) writes:
> To achieve disk write-order independence is probably not a worthwhile
> goal, but for systems that cannot provide strict write ordering (e.g.,
> most PCs) it would be helpful to be able to detect that the database
> has become corrupted. In Vadim's ex
On Fri, Jan 12, 2001 at 04:38:37PM -0800, Mikheev, Vadim wrote:
> Example.
> 1. Tuple was inserted into index.
> 2. Looking for free buffer bufmgr decides to write index block.
> 3. Following WAL core rule bufmgr first calls XLogFlush() to write
>and fsync log record related to index tuple ins
On Fri, Jan 12, 2001 at 11:30:30PM -0500, Tom Lane wrote:
> >> AFAICS, disk-block CRCs do not guard against mishaps involving intended
> >> writes. They will help guard against data corruption that might creep
> >> in due to outside factors, however.
>
> > Right.
>
> Given that we seem to hav
>> AFAICS, disk-block CRCs do not guard against mishaps involving intended
>> writes. They will help guard against data corruption that might creep
>> in due to outside factors, however.
> Right.
Given that we seem to have agreed on that, I withdraw my complaint about
disk-block-CRC not being
Nathan Myers wrote:
>
> It wouldn't help you recover, but you would be able to report that
> you cannot recover.
While this could help decting hardware problems, you still won't be able
to detect some (many) memory errors because the CRC will be calculated
on the already corrupted data.
Of cour
> > If log record was not really flushed on disk in 3. but
> > on-disk image of index block was updated in 4. and system
> > crashed after this then after restart recovery you'll have
> > unlawful index tuple pointing to where? Who knows!
> > No guarantee that corresponding heap tuple was flushed
On Fri, Jan 12, 2001 at 04:10:36PM -0800, Alfred Perlstein wrote:
> Nathan Myers <[EMAIL PROTECTED]> [010112 15:49] wrote:
> >
> > Obviously it's better to configure the disk so that it doesn't
> > lie about what's been written.
>
> I thought WAL+fsync wasn't supposed to allow this to happen?
It
"Mikheev, Vadim" <[EMAIL PROTECTED]> writes:
> If log record was not really flushed on disk in 3. but on-disk image of
> index block was updated in 4. and system crashed after this then after
> restart recovery you'll have unlawful index tuple pointing to where?
> Who knows! No guarantee that corr
> > How? The scenario Vadim is pointing out is where the disk
> > drive writes a changed data block in advance of the WAL log
> > entry describing the change. Then power drops and the WAL
> > entry never gets made. At restart, how will you realize that
> > that data block now contains data you d
* Nathan Myers <[EMAIL PROTECTED]> [010112 15:49] wrote:
> On Fri, Jan 12, 2001 at 06:06:21PM -0500, Tom Lane wrote:
> > [EMAIL PROTECTED] (Nathan Myers) writes:
> > >> "Changes must be logged *before* changed data pages written".
> > >> If this rule will be broken then data files will be
On Fri, Jan 12, 2001 at 06:06:21PM -0500, Tom Lane wrote:
> [EMAIL PROTECTED] (Nathan Myers) writes:
> >> "Changes must be logged *before* changed data pages written".
> >> If this rule will be broken then data files will be inconsistent
> >> after crash recovery and you will not notic
> > It wouldn't help you recover, but you would be able to report that
> > you cannot recover.
>
> How? The scenario Vadim is pointing out is where the disk
> drive writes a changed data block in advance of the WAL log entry
> describing the change. Then power drops and the WAL entry never gets
[EMAIL PROTECTED] (Nathan Myers) writes:
>> "Changes must be logged *before* changed data pages written".
>> If this rule will be broken then data files will be inconsistent
>> after crash recovery and you will not notice this, w/wo CRC in
>> data blocks.
You can include
On Fri, Jan 12, 2001 at 02:16:07PM -0800, Mikheev, Vadim wrote:
> > > You know - this is *core* assumption. If drive lies about this then
> > > *nothing* will help you. Do you remember core rule of WAL?
> > > "Changes must be logged *before* changed data pages written".
> > > If this rule will be
> > You know - this is *core* assumption. If drive lies about this then
> > *nothing* will help you. Do you remember core rule of WAL?
> > "Changes must be logged *before* changed data pages written".
> > If this rule will be broken then data files will be inconsistent
> > after crash recovery and
On Fri, Jan 12, 2001 at 12:35:14PM -0800, Nathan Myers wrote:
> Vadim wrote:
> > What do you mean by "external agents"?
>
> External agents include RAM bit drops and noise on cables when
> blocks are (read and re-) written. Every time data is moved,
> there is a chance of an undetected error be
On Fri, Jan 12, 2001 at 01:07:56PM -0800, Mikheev, Vadim wrote:
> > > But physical log recovery will rewrite every page that was changed
> > > after last checkpoint, thus this is not an issue anymore.
> >
> > No. That assumes that when the drive _says_ the block is written,
> > it is really on
> > But physical log recovery will rewrite every page that was changed
> > after last checkpoint, thus this is not an issue anymore.
>
> No. That assumes that when the drive _says_ the block is written,
> it is really on the disk. That is not true for IDE drives. It is
> true for SCSI drives
Vadim wrote:
> Tom wrote:
> > Bruce wrote:
> > > ... If the CRC on
> > > the WAL log checks for errors that are not checked anywhere else,
> > > then fine, but I thought disk CRC would just duplicate the I/O
> > > subsystem/disk.
> >
> > A disk-block CRC would detect partially written blocks (ie,
> A disk-block CRC would detect partially written blocks (ie, power drops
> after disk has written M of the N sectors in a block). The disk's own
> checks will NOT consider this condition a failure.
But physical log recovery will rewrite every page that was changed
after last checkpoint, thus t
> > No, I thought we agreed disk block CRC was way overkill. If the CRC on
> > the WAL log checks for errors that are not checked anywhere else, then
> > fine, but I thought disk CRC would just duplicate the I/O subsystem/disk
> > checks.
>
> A disk-block CRC would detect partially written block
At 21:55 11/01/01 -0500, Tom Lane wrote:
>
>Oh. I thought we'd agreed that a CRC on each stored disk block would
>be a good idea as well. I take it you didn't do that.
>
>Do we want to consider doing this (and forcing another initdb)?
>Or shall we say "too late for 7.1"?
>
I thought it was comi
Bruce Momjian <[EMAIL PROTECTED]> writes:
>> Oh. I thought we'd agreed that a CRC on each stored disk block would
>> be a good idea as well. I take it you didn't do that.
> No, I thought we agreed disk block CRC was way overkill. If the CRC on
> the WAL log checks for errors that are not check
> "Mikheev, Vadim" <[EMAIL PROTECTED]> writes:
> >> Actually, I'd expect the CRC check to catch an all-zeroes page (if
> >> it fails to complain, then you misimplemented the CRC), so that would
> >> be the place to deal with it now.
>
> > I've used standard CRC32 implementation you pointed me to
"Mikheev, Vadim" <[EMAIL PROTECTED]> writes:
>> Actually, I'd expect the CRC check to catch an all-zeroes page (if
>> it fails to complain, then you misimplemented the CRC), so that would
>> be the place to deal with it now.
> I've used standard CRC32 implementation you pointed me to -:)
> But CR
On Thu, Dec 07, 2000 at 12:22:12PM -0800, Mikheev, Vadim wrote:
> > > That's why an end marker must follow all valid records.
> ...
> >
> > That requires an extra out-of-sequence write.
>
> Yes, and also increase probability to corrupt already committed
> to log data.
>
> > (I'd also like to
On Thu, Dec 07, 2000 at 12:22:12PM -0800, Mikheev, Vadim wrote:
> > > That's why an end marker must follow all valid records.
> > That requires an extra out-of-sequence write.
> Yes, and also increase probability to corrupt already committed
> to log data.
Are you referring to the case where t
On Thu, Dec 07, 2000 at 12:25:41PM -0800, Nathan Myers wrote:
> That requires an extra out-of-sequence write.
Ayup!
> Generally, there are no guarantees, only reasonable expectations.
I would differ, but that's irrelevant.
> A 64-bit CRC would give sufficient confidence...
This is part of wh
> > (I'd also like to see CRCs on all the table blocks as well; is there
> > a place to put them?)
>
> Do we need it? "physical log" feature suggested by Andreas will protect
> us from non atomic data block writes.
CRCs are neccessary because of glitches, hardware failures, operating system
bugs,
P.S.: I would volunteer to integrate CRC routines into postgres if somebody
points me in the right direction in the source code.
Horst
> > That's why an end marker must follow all valid records.
...
>
> That requires an extra out-of-sequence write.
Yes, and also increase probability to corrupt already committed
to log data.
> (I'd also like to see CRCs on all the table blocks as well; is there
> a place to put them?)
Do we
On Wed, Dec 06, 2000 at 06:53:37PM -0600, Bruce Guenter wrote:
> On Wed, Dec 06, 2000 at 11:08:00AM -0800, Nathan Myers wrote:
> > On Wed, Dec 06, 2000 at 11:49:10AM -0600, Bruce Guenter wrote:
> > >
> > > I don't know how pgsql does it, but the only safe way I know of
> > > is to include an "end
On Wed, Dec 06, 2000 at 11:08:00AM -0800, Nathan Myers wrote:
> On Wed, Dec 06, 2000 at 11:49:10AM -0600, Bruce Guenter wrote:
> > On Wed, Dec 06, 2000 at 11:15:26AM -0500, Tom Lane wrote:
> > > How exactly *do* we determine where the end of the valid log data is,
> > > anyway?
> >
> > I don't kn
On Wed, Dec 06, 2000 at 11:49:10AM -0600, Bruce Guenter wrote:
> On Wed, Dec 06, 2000 at 11:15:26AM -0500, Tom Lane wrote:
> > Zeugswetter Andreas SB <[EMAIL PROTECTED]> writes:
> > > Yes, but there would need to be a way to verify the last page or
> > > record from txlog when running on crap hard
On Wed, Dec 06, 2000 at 12:29:00PM +0100, Zeugswetter Andreas SB wrote:
>
> > Why should we do this? I'm not going to replay parts individually,
> > I'm going to write entire pages to OS cache and than apply changes
> > to them. Recovery is considered as succeeded after server is ensured
> > that
41 matches
Mail list logo
