Re: [HACKERS] ginInsertCleanup called from vacuum could still miss tuples to be deleted

2017-11-24 Thread Peter Geoghegan
On Fri, Nov 24, 2017 at 6:09 PM, Jeff Janes wrote: >> We see this for the entry tree (no deletion is possible in the first >> place), and we also see it for the posting tree (the dance with >> inserters having a pin on the root, and so on). Not mentioning why >> pending list recycling is safe in t

Re: [HACKERS] ginInsertCleanup called from vacuum could still miss tuples to be deleted

2017-11-24 Thread Jeff Janes
On Thu, Nov 16, 2017 at 2:43 PM, Peter Geoghegan wrote: > On Thu, Nov 16, 2017 at 2:16 PM, Jeff Janes wrote: > > The only reference to super-exclusive lock in > src/backend/access/gin/README, > > that I can find, is about posting trees, not pending lists. Can you > quote > > or give line number

Re: [HACKERS] ginInsertCleanup called from vacuum could still miss tuples to be deleted

2017-11-24 Thread Jeff Janes
On Thu, Nov 16, 2017 at 12:29 PM, Robert Haas wrote: > On Thu, Nov 16, 2017 at 7:08 AM, Masahiko Sawada > wrote: > > Agreed, that's better. Attached updated patch. > > Also I've added this to the next CF so as not to forget. > > Committed and back-patched. While I'm fairly sure this is a correc

Re: [HACKERS] ginInsertCleanup called from vacuum could still miss tuples to be deleted

2017-11-18 Thread Peter Geoghegan
On Thu, Nov 16, 2017 at 2:16 PM, Jeff Janes wrote: > e2c79e14 was to fix a pre-existing bug, but probably e9568083 made that bug > easier to hit than it was before. (Which is not to say that e9568083 can't > contain bugs of its own, of course) I get that. I think that the same thing may have ha

Re: [HACKERS] ginInsertCleanup called from vacuum could still miss tuples to be deleted

2017-11-16 Thread Peter Geoghegan
On Thu, Nov 16, 2017 at 2:16 PM, Jeff Janes wrote: > The only reference to super-exclusive lock in src/backend/access/gin/README, > that I can find, is about posting trees, not pending lists. Can you quote > or give line numbers of the section you are referring to? That's the section I was refer

Re: [HACKERS] ginInsertCleanup called from vacuum could still miss tuples to be deleted

2017-11-16 Thread Jeff Janes
On Mon, Nov 13, 2017 at 8:08 PM, Peter Geoghegan wrote: > On Mon, Nov 13, 2017 at 6:56 PM, Masahiko Sawada > wrote: > >/* > > * We would like to prevent concurrent cleanup process. For that we > will > > * lock metapage in exclusive mode using LockPage() call. Nobody other > > *

Re: [HACKERS] ginInsertCleanup called from vacuum could still miss tuples to be deleted

2017-11-16 Thread Peter Geoghegan
On Thu, Nov 16, 2017 at 12:49 PM, Peter Geoghegan wrote: > But there is only ever one page locked, the meta-page. And, it's > always an ExclusiveLock. I don't see much use for deadlock avoidance. I meant deadlock detection. -- Peter Geoghegan

Re: [HACKERS] ginInsertCleanup called from vacuum could still miss tuples to be deleted

2017-11-16 Thread Peter Geoghegan
On Thu, Nov 16, 2017 at 12:44 PM, Robert Haas wrote: > I would like to get rid of that LockPage() call, for sure, because > it's problematic in terms of allowing writes in parallel mode. > However, I think the reason here is the same as why the hash AM used > to use them. If you use a buffer lock

Re: [HACKERS] ginInsertCleanup called from vacuum could still miss tuples to be deleted

2017-11-16 Thread Robert Haas
On Mon, Nov 13, 2017 at 8:48 PM, Peter Geoghegan wrote: > One thing that really bothers me about commit e2c79e14 is that > LockPage() is called, not LockBuffer(). GIN had no LockPage() calls > before that commit, and is now the only code in the entire system that > calls LockPage()/ConditionalLock

Re: [HACKERS] ginInsertCleanup called from vacuum could still miss tuples to be deleted

2017-11-16 Thread Peter Geoghegan
On Thu, Nov 16, 2017 at 12:29 PM, Robert Haas wrote: > I am also clear on what the consequences of this bug are. It seems > like it could harm insert performance by making us wait when we > shouldn't, but can it cause corruption? That I'm not sure about. If > there's already a cleanup of the pe

Re: [HACKERS] ginInsertCleanup called from vacuum could still miss tuples to be deleted

2017-11-16 Thread Robert Haas
On Thu, Nov 16, 2017 at 7:08 AM, Masahiko Sawada wrote: > Agreed, that's better. Attached updated patch. > Also I've added this to the next CF so as not to forget. Committed and back-patched. While I'm fairly sure this is a correct fix, post-commit review from someone who knows GIN better than I

Re: [HACKERS] ginInsertCleanup called from vacuum could still miss tuples to be deleted

2017-11-16 Thread Masahiko Sawada
On Thu, Nov 16, 2017 at 11:24 AM, Robert Haas wrote: > On Mon, Nov 13, 2017 at 3:25 AM, Masahiko Sawada > wrote: >> In ginInsertCleanup(), we lock the GIN meta page by LockPage and could >> wait for the concurrent cleaning up process if stats == NULL. And the >> source code comment says that thi

Re: [HACKERS] ginInsertCleanup called from vacuum could still miss tuples to be deleted

2017-11-15 Thread Robert Haas
On Mon, Nov 13, 2017 at 3:25 AM, Masahiko Sawada wrote: > In ginInsertCleanup(), we lock the GIN meta page by LockPage and could > wait for the concurrent cleaning up process if stats == NULL. And the > source code comment says that this happen is when ginINsertCleanup is > called by [auto]vacuum/

Re: [HACKERS] ginInsertCleanup called from vacuum could still miss tuples to be deleted

2017-11-13 Thread Peter Geoghegan
On Mon, Nov 13, 2017 at 6:56 PM, Masahiko Sawada wrote: >/* > * We would like to prevent concurrent cleanup process. For that we will > * lock metapage in exclusive mode using LockPage() call. Nobody other > * will use that lock for metapage, so we keep possibility of concurrent >

Re: [HACKERS] ginInsertCleanup called from vacuum could still miss tuples to be deleted

2017-11-13 Thread Masahiko Sawada
On Tue, Nov 14, 2017 at 10:48 AM, Peter Geoghegan wrote: > On Mon, Nov 13, 2017 at 5:07 PM, Masahiko Sawada > wrote: >>> I've been suspicious of that commit (and related commits) for a while >>> now [1]. I think that it explains a couple of different problem >>> reports that we have seen. >> >>

Re: [HACKERS] ginInsertCleanup called from vacuum could still miss tuples to be deleted

2017-11-13 Thread Michael Paquier
On Tue, Nov 14, 2017 at 10:52 AM, Eric Lam wrote: > Please unsubscribe me. (Please do not hijack the threads) Help yourself: https://www.postgresql.org/community/lists/subscribe/ -- Michael

Re: [HACKERS] ginInsertCleanup called from vacuum could still miss tuples to be deleted

2017-11-13 Thread Eric Lam
Hi, Please unsubscribe me. thanksEric On Tuesday, November 14, 2017, 2:02:04 AM GMT+8, Peter Geoghegan wrote: On Mon, Nov 13, 2017 at 12:25 AM, Masahiko Sawada wrote: > Commit e2c79e14 prevented multiple cleanup process for pending list in > GIN index. But I think that there is still

Re: [HACKERS] ginInsertCleanup called from vacuum could still miss tuples to be deleted

2017-11-13 Thread Peter Geoghegan
On Mon, Nov 13, 2017 at 5:07 PM, Masahiko Sawada wrote: >> I've been suspicious of that commit (and related commits) for a while >> now [1]. I think that it explains a couple of different problem >> reports that we have seen. > > Yeah, the problem here is that vacuum and analyze don't acquire a >

Re: [HACKERS] ginInsertCleanup called from vacuum could still miss tuples to be deleted

2017-11-13 Thread Masahiko Sawada
On Tue, Nov 14, 2017 at 3:01 AM, Peter Geoghegan wrote: > On Mon, Nov 13, 2017 at 12:25 AM, Masahiko Sawada > wrote: >> Commit e2c79e14 prevented multiple cleanup process for pending list in >> GIN index. But I think that there is still possibility that vacuum >> could miss tuples to be deleted

Re: [HACKERS] ginInsertCleanup called from vacuum could still miss tuples to be deleted

2017-11-13 Thread Peter Geoghegan
On Mon, Nov 13, 2017 at 12:25 AM, Masahiko Sawada wrote: > Commit e2c79e14 prevented multiple cleanup process for pending list in > GIN index. But I think that there is still possibility that vacuum > could miss tuples to be deleted if someone else is cleaning up the > pending list. I've been sus