Re: [HACKERS] Block level concurrency during recovery

2008-11-03 Thread Tom Lane
Simon Riggs <[EMAIL PROTECTED]> writes: > Are you happy with my optimisation that if a page needs to be read in, > we just skip it and pretend we did read-pin-unpin on it? If it's not in buffers then it cannot be pinned, so I concur that that is safe. regards, tom lane --

Re: [HACKERS] Block level concurrency during recovery

2008-11-03 Thread Simon Riggs
On Mon, 2008-11-03 at 10:07 -0500, Tom Lane wrote: > Simon Riggs <[EMAIL PROTECTED]> writes: > > VACUUM with a btree index proceeds like this: > > 1. Scan table > > 2. Remove rows from btree identified in (1) > > 3. Remove rows from heap identified in (1) > > > The purpose of the additional locki

Re: [HACKERS] Block level concurrency during recovery

2008-11-03 Thread Tom Lane
Simon Riggs <[EMAIL PROTECTED]> writes: > VACUUM with a btree index proceeds like this: > 1. Scan table > 2. Remove rows from btree identified in (1) > 3. Remove rows from heap identified in (1) > The purpose of the additional locking requirements during (2) for btrees > is to ensure that we do no

Re: [HACKERS] Block level concurrency during recovery

2008-11-03 Thread Simon Riggs
On Thu, 2008-10-23 at 09:57 +0100, Simon Riggs wrote: > On Thu, 2008-10-23 at 09:09 +0300, Heikki Linnakangas wrote: > > > However, we require that in b-tree vacuum, you take a cleanup lock on > > *every* leaf page of the index, not only those that you modify. That's a > > problem, because ther

Re: [HACKERS] Block level concurrency during recovery

2008-10-23 Thread Simon Riggs
On Thu, 2008-10-23 at 20:24 +0300, Heikki Linnakangas wrote: > Simon Riggs wrote: > > On Thu, 2008-10-23 at 19:21 +0300, Heikki Linnakangas wrote: > >> You have other issues with those still, namely that > >> table locks are not yet taken appropriately, so I'm not sure if this is > >> worth worr

Re: [HACKERS] Block level concurrency during recovery

2008-10-23 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2008-10-23 at 19:21 +0300, Heikki Linnakangas wrote: You have other issues with those still, namely that table locks are not yet taken appropriately, so I'm not sure if this is worth worrying about until that's done. Please explain. If you know of a correctness issu

Re: [HACKERS] Block level concurrency during recovery

2008-10-23 Thread Simon Riggs
On Thu, 2008-10-23 at 19:21 +0300, Heikki Linnakangas wrote: > Simon Riggs wrote: > > It would also be possible to introduce a special tweak there which is > > that if the block is not in cache, don't read it in at all. If its not > > in cache we know that nobody has a pin on it, so don't need to

Re: [HACKERS] Block level concurrency during recovery

2008-10-23 Thread Heikki Linnakangas
Simon Riggs wrote: It would also be possible to introduce a special tweak there which is that if the block is not in cache, don't read it in at all. If its not in cache we know that nobody has a pin on it, so don't need to read it in just to say "got the lock". That icing for later. Heh, that's

Re: [HACKERS] Block level concurrency during recovery

2008-10-23 Thread Simon Riggs
On Thu, 2008-10-23 at 09:09 +0300, Heikki Linnakangas wrote: > However, we require that in b-tree vacuum, you take a cleanup lock on > *every* leaf page of the index, not only those that you modify. That's a > problem, because there's no trace of such pages in the WAL. OK, good. Thanks for the

Re: [HACKERS] Block level concurrency during recovery

2008-10-22 Thread Heikki Linnakangas
Simon Riggs wrote: I'm undecided as to whether this is too much code or too little. I'm somewhat uncertain as to the locking requirements for the physical scan during a vacuum. I've worked out various options if we need to change this. For the B-tree, an exclusive lock is enough to modify the c

Re: [HACKERS] Block level concurrency during recovery

2008-10-22 Thread Simon Riggs
On Mon, 2008-10-20 at 13:54 +0100, Simon Riggs wrote: > I'm looking at how to make queries safe during recovery, in the presence > of concurrent changes to blocks. In particular, concurrent removal of > rows that might be read by queries. > > My thinking is > * we ignore LockRelationForExtensio

Re: [HACKERS] Block level concurrency during recovery

2008-10-20 Thread Teodor Sigaev
But does holding cleanup lock on root prevent an in-progress Insert from changing non-root pages? I assume so, just not sure how. Yes, because insertion process doesn't unpin root page until insert will done. -- Teodor Sigaev E-mail: [EMAIL PROTECTED]

Re: [HACKERS] Block level concurrency during recovery

2008-10-20 Thread Simon Riggs
On Mon, 2008-10-20 at 20:12 +0400, Teodor Sigaev wrote: > > I don't understand why in ginVacuumPostingTreeLeaves() we lock only the > > root page for Cleanup and no others. Why do we need to hold Cleanup lock > > on the root? If the index is concurrent safe for existing scans, why is > > it not sa

Re: [HACKERS] Block level concurrency during recovery

2008-10-20 Thread Teodor Sigaev
I don't understand why in ginVacuumPostingTreeLeaves() we lock only the root page for Cleanup and no others. Why do we need to hold Cleanup lock on the root? If the index is concurrent safe for existing scans, why is it not safe for new scans also? And the converse: if it is not safe for new scans

Re: [HACKERS] Block level concurrency during recovery

2008-10-20 Thread Simon Riggs
On Mon, 2008-10-20 at 18:28 +0400, Teodor Sigaev wrote: > > * For GIN indexes, we appear to not hold a Cleanup lock during > > vacuuming, except on root page. That stops new scans from starting, but > > it doesn't prevent progress of concurrent scans. Doesn't look correct to > > me... so not sure

Re: [HACKERS] Block level concurrency during recovery

2008-10-20 Thread Teodor Sigaev
* For GIN indexes, we appear to not hold a Cleanup lock during vacuuming, except on root page. That stops new scans from starting, but it doesn't prevent progress of concurrent scans. Doesn't look correct to me... so not sure what strength lock to acquire in each case. Probably Why do you think s

Re: [HACKERS] Block level concurrency during recovery

2008-10-20 Thread Simon Riggs
On Mon, 2008-10-20 at 14:23 +0100, Gregory Stark wrote: > Simon Riggs <[EMAIL PROTECTED]> writes: > > > * re-create the Cleanup lock on blocks, when the original operation was > > performed while a Cleanup lock was held. > > > > So the plan is to introduce a new XLogLockBufferForCleanup() functio

Re: [HACKERS] Block level concurrency during recovery

2008-10-20 Thread Gregory Stark
Simon Riggs <[EMAIL PROTECTED]> writes: > * re-create the Cleanup lock on blocks, when the original operation was > performed while a Cleanup lock was held. > > So the plan is to introduce a new XLogLockBufferForCleanup() function > and then use it in all places where a cleanup lock was held duri

[HACKERS] Block level concurrency during recovery

2008-10-20 Thread Simon Riggs
I'm looking at how to make queries safe during recovery, in the presence of concurrent changes to blocks. In particular, concurrent removal of rows that might be read by queries. My thinking is * we ignore LockRelationForExtension(). Normal queries never request it. All new blocks were created wi