Re: REINDEX CONCURRENTLY 2.0

2019-04-16 Thread Michael Paquier
On Tue, Apr 16, 2019 at 08:50:31AM +0200, Peter Eisentraut wrote: > Looks good to me. Thanks, committed. If there are additional discussions on various points of the feature, let's move to a new thread please. This one has been already extensively used ;) -- Michael signature.asc Description:

Re: REINDEX CONCURRENTLY 2.0

2019-04-15 Thread Peter Eisentraut
On 2019-04-16 08:19, Michael Paquier wrote: > On Fri, Apr 12, 2019 at 12:11:12PM +0100, Dagfinn Ilmari Mannsåker wrote: >> I don't have any comments on the code (but the test looks sensible, it's >> the same trick I used to discover the issue in the first place). > > After thinking some more on it

Re: REINDEX CONCURRENTLY 2.0

2019-04-15 Thread Michael Paquier
On Fri, Apr 12, 2019 at 12:11:12PM +0100, Dagfinn Ilmari Mannsåker wrote: > I don't have any comments on the code (but the test looks sensible, it's > the same trick I used to discover the issue in the first place). After thinking some more on it, this behavior looks rather sensible to me. Are th

Re: REINDEX CONCURRENTLY 2.0

2019-04-12 Thread Dagfinn Ilmari Mannsåker
Michael Paquier writes: > So... I am coming up with the patch attached. I have introduced some > tests using a trick with CIC to have an invalid index to work on. I don't have any comments on the code (but the test looks sensible, it's the same trick I used to discover the issue in the first p

Re: REINDEX CONCURRENTLY 2.0

2019-04-11 Thread Michael Paquier
On Fri, Apr 12, 2019 at 08:44:18AM +0200, Peter Eisentraut wrote: > Looks good, committed. Thanks for committing! -- Michael signature.asc Description: PGP signature

Re: REINDEX CONCURRENTLY 2.0

2019-04-11 Thread Peter Eisentraut
On 2019-04-11 05:59, Michael Paquier wrote: > On Tue, Apr 09, 2019 at 03:50:27PM +0900, Michael Paquier wrote: >> And here is the patch to address this issue. It happens that a bit >> more than the dependency switch was lacking here: >> - At swap time, we need to have the new index definition trac

Re: REINDEX CONCURRENTLY 2.0

2019-04-11 Thread Michael Paquier
On Thu, Apr 11, 2019 at 09:49:47AM -0400, Alvaro Herrera wrote: > Hmm, I suppose that makes sense for REINDEX TABLE or anything that > reindexes more than one index, but if you do REINDEX INDEX surely it > is reasonable to allow the case? Yes, we could revisit the REINDEX INDEX portion of the deci

Re: REINDEX CONCURRENTLY 2.0

2019-04-11 Thread Alvaro Herrera
On 2019-Apr-11, Michael Paquier wrote: > > I was going to just remove the caveat, but then I discovered that > > REINDEX CONCURRENTLY doesn't work on INVALID indexes (why?). > > This is a wanted choice. The index built in parallel of the existing > one during a concurrent reindex is marked inval

Re: REINDEX CONCURRENTLY 2.0

2019-04-11 Thread Michael Paquier
On Thu, Apr 11, 2019 at 11:21:29AM +0100, Dagfinn Ilmari Mannsåker wrote: > I noticed that the docs for how to recover from a failed CREATE INDEX > CONCURRENTLY say that «REINDEX does not support concurrent builds», > which is no longer true. Good catch. I'll apply that in a couple of hours. > I

Re: REINDEX CONCURRENTLY 2.0

2019-04-11 Thread Dagfinn Ilmari Mannsåker
Peter Eisentraut writes: > On 2019-03-28 09:07, Sergei Kornilov wrote: >> Unfortunately patch does not apply due recent commits. Any chance >> this can be fixed (and even committed in pg12)? > > Committed :) I noticed that the docs for how to recover from a failed CREATE INDEX CONCURRENTLY say t

Re: REINDEX CONCURRENTLY 2.0

2019-04-10 Thread Michael Paquier
On Tue, Apr 09, 2019 at 03:50:27PM +0900, Michael Paquier wrote: > And here is the patch to address this issue. It happens that a bit > more than the dependency switch was lacking here: > - At swap time, we need to have the new index definition track > relispartition from the old index. > - Again

Re: REINDEX CONCURRENTLY 2.0

2019-04-08 Thread Michael Paquier
On Mon, Apr 01, 2019 at 03:43:43PM +0900, Michael Paquier wrote: > And I would have expected concur_reindex_part1v1_c1_idx to still be > part of the partition tree. I think that the issue is in > index_concurrently_create_copy() where we create the new index with > index_create() without setting p

Re: REINDEX CONCURRENTLY 2.0

2019-03-31 Thread Michael Paquier
On Sat, Mar 30, 2019 at 11:56:27AM +0100, Peter Eisentraut wrote: > The attached patch fixes this. The issue was that we didn't move all > dependencies from the index (only in the other direction). Maybe that > was sufficient when the patch was originally written, before partitioned > indexes. H

Re: REINDEX CONCURRENTLY 2.0

2019-03-31 Thread Michael Paquier
On Sat, Mar 30, 2019 at 11:56:27AM +0100, Peter Eisentraut wrote: > The attached patch fixes this. The issue was that we didn't move all > dependencies from the index (only in the other direction). Maybe that > was sufficient when the patch was originally written, before partitioned > indexes. H

Re: REINDEX CONCURRENTLY 2.0

2019-03-30 Thread Peter Eisentraut
On 2019-03-29 16:10, Shinoda, Noriyoshi (PN Japan A&PS Delivery) wrote: > postgres=> CREATE TABLE part1(c1 INT) PARTITION BY RANGE(c1); > CREATE TABLE > postgres=> CREATE TABLE part1v1 PARTITION OF part1 FOR VALUES FROM (0) TO > (100); > CREATE TABLE > postgres=> CREATE INDEX idx1_part1 ON part1(c

Re: REINDEX CONCURRENTLY 2.0

2019-03-29 Thread Michael Paquier
On Fri, Mar 29, 2019 at 08:48:03AM -0700, Andres Freund wrote: > Yes, it increases the total runtime quite considerably. And it adds new > failure modes with partially built invalid indexes hanging around that > need to be dropped manually. On top of that CONCURRENTLY needs multiple transactions t

Re: REINDEX CONCURRENTLY 2.0

2019-03-29 Thread Michael Paquier
On Fri, Mar 29, 2019 at 03:10:23PM +, Shinoda, Noriyoshi (PN Japan A&PS Delivery) wrote: > I tried this great feature for partition index. > The first time the REINDEX TABLE CONCURRENTLY statement is executed > to the partition, then an error occurs. Yes, that's a problem. I am adding an op

Re: REINDEX CONCURRENTLY 2.0

2019-03-29 Thread Peter Eisentraut
On 2019-03-29 17:01, Justin Pryzby wrote: > On Fri, Mar 29, 2019 at 03:53:05PM +, Bossart, Nathan wrote: >> I noticed a very small typo in the documentation for this feature. > > I submit a bunch more changes for consideration, attached. fixed, thanks -- Peter Eisentraut http:/

Re: REINDEX CONCURRENTLY 2.0

2019-03-29 Thread Peter Eisentraut
On 2019-03-29 16:53, Bossart, Nathan wrote: > I noticed a very small typo in the documentation for this feature. fixed -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: REINDEX CONCURRENTLY 2.0

2019-03-29 Thread Justin Pryzby
On Fri, Mar 29, 2019 at 03:53:05PM +, Bossart, Nathan wrote: > I noticed a very small typo in the documentation for this feature. I submit a bunch more changes for consideration, attached. >From dafdb15fb3e7c69de82a2206c9bf07588b5665ce Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Fri, 2

Re: REINDEX CONCURRENTLY 2.0

2019-03-29 Thread Bossart, Nathan
I noticed a very small typo in the documentation for this feature. diff --git a/doc/src/sgml/ref/reindex.sgml b/doc/src/sgml/ref/reindex.sgml index ccabb330cb..e45bf86c8d 100644 --- a/doc/src/sgml/ref/reindex.sgml +++ b/doc/src/sgml/ref/reindex.sgml @@ -349,7 +349,7 @@ REINDEX [ ( VERBOSE ) ] { IN

Re: REINDEX CONCURRENTLY 2.0

2019-03-29 Thread Andres Freund
Hi, On 2019-03-29 11:47:10 -0400, Robert Treat wrote: > On Fri, Mar 29, 2019 at 3:28 AM Peter Eisentraut > wrote: > > > > On 2019-03-28 09:07, Sergei Kornilov wrote: > > > Unfortunately patch does not apply due recent commits. Any chance this > > > can be fixed (and even committed in pg12)? > >

Re: REINDEX CONCURRENTLY 2.0

2019-03-29 Thread Robert Treat
On Fri, Mar 29, 2019 at 3:28 AM Peter Eisentraut wrote: > > On 2019-03-28 09:07, Sergei Kornilov wrote: > > Unfortunately patch does not apply due recent commits. Any chance this can > > be fixed (and even committed in pg12)? > > Committed :) > Given this has been committed I've probably missed

RE: REINDEX CONCURRENTLY 2.0

2019-03-29 Thread Shinoda, Noriyoshi (PN Japan A&PS Delivery)
ent: Friday, March 29, 2019 6:21 PM To: Peter Eisentraut Cc: Sergei Kornilov ; pgsql-hackers@lists.postgresql.org Subject: Re: REINDEX CONCURRENTLY 2.0 On Fri, Mar 29, 2019 at 09:13:35AM +0100, Peter Eisentraut wrote: > So, we're getting buildfarm failures, only with clang. I can > reproduc

Re: REINDEX CONCURRENTLY 2.0

2019-03-29 Thread Michael Paquier
On Fri, Mar 29, 2019 at 09:13:35AM +0100, Peter Eisentraut wrote: > So, we're getting buildfarm failures, only with clang. I can reproduce > those (with clang). Indeed, I can reproduce the failures using -O2 with clang. I am wondering if we are not missing a volatile flag somewhere and that some

Re: REINDEX CONCURRENTLY 2.0

2019-03-29 Thread Peter Eisentraut
On 2019-03-29 09:13, Peter Eisentraut wrote: > On 2019-03-29 09:04, Michael Paquier wrote: >> On Fri, Mar 29, 2019 at 10:39:23AM +0300, Sergei Kornilov wrote: >>> wow! Congratulations! This was very long way >>> >>> my favorite pg12 feature >> >> So this has been committed, nice! Thanks a lot to a

Re: REINDEX CONCURRENTLY 2.0

2019-03-29 Thread Peter Eisentraut
On 2019-03-29 09:04, Michael Paquier wrote: > On Fri, Mar 29, 2019 at 10:39:23AM +0300, Sergei Kornilov wrote: >> wow! Congratulations! This was very long way >> >> my favorite pg12 feature > > So this has been committed, nice! Thanks a lot to all for keeping > alive this patch over the ages, wit

Re: REINDEX CONCURRENTLY 2.0

2019-03-29 Thread Michael Paquier
On Fri, Mar 29, 2019 at 10:39:23AM +0300, Sergei Kornilov wrote: > wow! Congratulations! This was very long way > > my favorite pg12 feature So this has been committed, nice! Thanks a lot to all for keeping alive this patch over the ages, with particular thanks to Andreas and Peter. -- Michael

Re: REINDEX CONCURRENTLY 2.0

2019-03-29 Thread Sergei Kornilov
>>  Unfortunately patch does not apply due recent commits. Any chance this can >> be fixed (and even committed in pg12)? > > Committed :) wow! Congratulations! This was very long way my favorite pg12 feature regards, Sergei

Re: REINDEX CONCURRENTLY 2.0

2019-03-29 Thread Peter Eisentraut
On 2019-03-28 09:07, Sergei Kornilov wrote: > Unfortunately patch does not apply due recent commits. Any chance this can be > fixed (and even committed in pg12)? Committed :) -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training

Re: REINDEX CONCURRENTLY 2.0

2019-03-28 Thread Sergei Kornilov
Hi Unfortunately patch does not apply due recent commits. Any chance this can be fixed (and even committed in pg12)? >>  And few questions: >>  - reindexdb has concurrently flag logic even in reindex_system_catalogs, >> but "reindex concurrently" can not reindex system catalog. Is this expected

Re: REINDEX CONCURRENTLY 2.0

2019-03-25 Thread Michael Paquier
On Mon, Mar 25, 2019 at 04:23:34PM +0100, Peter Eisentraut wrote: > Let's do it. :-) I am pretty sure that this has been said at least once since 2012. > I've gone over this patch a few more times. I've read all the > discussion since 2012 again and made sure all the issues were addressed. > I

Re: REINDEX CONCURRENTLY 2.0

2019-03-23 Thread Sergei Kornilov
Hello Yet another review of this patch from me... >An index build with the CONCURRENTLY option failed, > leaving >an invalid index. Such indexes are useless but it can be > - convenient to use REINDEX to rebuild them. Note that > - REINDEX will not perform a concurrent

Re: REINDEX CONCURRENTLY 2.0

2019-03-20 Thread Peter Eisentraut
On 2019-03-15 22:32, Michael Banck wrote: > I had a quick look at some of the comments and noticed some possible > nitpicky-level problems: Thanks, I've integrated these changes into my local branch. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Suppor

Re: REINDEX CONCURRENTLY 2.0

2019-03-15 Thread Michael Banck
Hi, Am Mittwoch, den 13.03.2019, 23:10 +0100 schrieb Peter Eisentraut: > Here is an updated patch. I had a quick look at some of the comments and noticed some possible nitpicky-level problems: > +/* > + * index_concurrently_set_dead > + * > + * Perform the last invalidation stage of DROP INDEX C

Re: REINDEX CONCURRENTLY 2.0

2019-03-13 Thread Peter Eisentraut
On 2019-03-13 15:13, Sergei Kornilov wrote: > Patch is marked as target version 12, but is inactive few weeks long. I think > many users want this feature and patch is in good shape. We have open > questions on this thread? > > Latest patch still can be aplied cleanly; it builds and pass tests.

Re: REINDEX CONCURRENTLY 2.0

2019-03-13 Thread Sergei Kornilov
Hello Patch is marked as target version 12, but is inactive few weeks long. I think many users want this feature and patch is in good shape. We have open questions on this thread? Latest patch still can be aplied cleanly; it builds and pass tests. regards, Sergei

Re: REINDEX CONCURRENTLY 2.0

2019-02-07 Thread Sergei Kornilov
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:tested, passed Hello Sorry for late response. I review latest patch version. I