Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-08-05 Thread Alvaro Herrera
On 2022-Jul-30, Tom Lane wrote: > BTW, quite aside from stability, is it really necessary for this test to > be so freakin' slow? florican for instance reports > > [12:54:07] t/033_replay_tsp_drops.pl ok 117840 ms ( 0.01 usr > 0.00 sys + 8.72 cusr 5.41 csys = 14.14 CPU) > > 0

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-08-03 Thread Thomas Munro
On Sun, Jul 31, 2022 at 11:17 AM Tom Lane wrote: > Thomas Munro writes: > > I noticed this is a 32 bit FBSD system. Is it running on UFS, perhaps > > on slow storage? Are soft updates enabled (visible as options in > > output of "mount")? > > It's an ancient (2006) mac mini with 5400RPM spinnin

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-08-02 Thread Thomas Munro
On Sun, Jul 31, 2022 at 3:46 PM Thomas Munro wrote: > On Sun, Jul 31, 2022 at 2:37 AM Tom Lane wrote: > > Alvaro Herrera writes: > > > WFM, pushed that way. > > > > Looks like conchuela is still intermittently unhappy. > > > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=conchuela&dt=

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-31 Thread Andres Freund
Hi, On 2022-07-30 10:37:55 -0400, Tom Lane wrote: > Alvaro Herrera writes: > > WFM, pushed that way. > > Looks like conchuela is still intermittently unhappy. > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=conchuela&dt=2022-07-30%2004%3A57%3A51 CI as well: https://cirrus-ci.com/ta

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-30 Thread Thomas Munro
On Sun, Jul 31, 2022 at 2:37 AM Tom Lane wrote: > Alvaro Herrera writes: > > WFM, pushed that way. > > Looks like conchuela is still intermittently unhappy. > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=conchuela&dt=2022-07-30%2004%3A57%3A51 And here's one from CI that failed on Li

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-30 Thread Tom Lane
Thomas Munro writes: > I noticed this is a 32 bit FBSD system. Is it running on UFS, perhaps > on slow storage? Are soft updates enabled (visible as options in > output of "mount")? It's an ancient (2006) mac mini with 5400RPM spinning rust. "mount" says /dev/ada0s2a on / (ufs, local, soft-upd

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-30 Thread Thomas Munro
On Sun, Jul 31, 2022 at 4:51 AM Tom Lane wrote: > BTW, quite aside from stability, is it really necessary for this test to > be so freakin' slow? florican for instance reports > > [12:43:38] t/025_stuck_on_old_timeline.pl ... ok49010 ms ( 0.00 usr > 0.00 sys + 3.64 cusr 2.49 csys = 6

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-30 Thread Tom Lane
I wrote: > Looks like conchuela is still intermittently unhappy. BTW, quite aside from stability, is it really necessary for this test to be so freakin' slow? florican for instance reports [12:43:38] t/025_stuck_on_old_timeline.pl ... ok49010 ms ( 0.00 usr 0.00 sys + 3.64 cusr 2.49 c

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-30 Thread Tom Lane
Alvaro Herrera writes: > WFM, pushed that way. Looks like conchuela is still intermittently unhappy. https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=conchuela&dt=2022-07-30%2004%3A57%3A51 regards, tom lane

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-29 Thread Alvaro Herrera
On 2022-Jul-29, Kyotaro Horiguchi wrote: > At Fri, 29 Jul 2022 11:27:01 +1200, Thomas Munro > wrote in > > Maybe it just needs a replication slot? I see: > > > > ERROR: requested WAL segment 00010003 has already been > > removed > > Agreed, I see the same. The same failure

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-28 Thread Kyotaro Horiguchi
At Fri, 29 Jul 2022 11:27:01 +1200, Thomas Munro wrote in > Maybe it just needs a replication slot? I see: > > ERROR: requested WAL segment 00010003 has already been > removed Agreed, I see the same. The same failure can be surely reproducible by inserting wal-switch+checkp

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-28 Thread Thomas Munro
On Fri, Jul 29, 2022 at 9:57 AM Tom Lane wrote: > Matthias van de Meent writes: > > I'd like to bring to your attention that the test that was introduced > > with 9e4f914b seem to be flaky in FreeBSD 13 in the CFBot builds: it > > sometimes times out while waiting for the secondary to catch up. O

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-28 Thread Tom Lane
Matthias van de Meent writes: > I'd like to bring to your attention that the test that was introduced > with 9e4f914b seem to be flaky in FreeBSD 13 in the CFBot builds: it > sometimes times out while waiting for the secondary to catch up. Or, > at least I think it does, and I'm not too familiar w

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-28 Thread Matthias van de Meent
On Wed, 27 Jul 2022 at 20:55, Alvaro Herrera wrote: > > Okay, I think I'm done with this. Here's v27 for the master branch, > where I fixed some comments as well as thinkos in the test script. > The ones on older branches aren't materially different, they just have > tonnes of conflicts resolved.

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-27 Thread Alvaro Herrera
Okay, I think I'm done with this. Here's v27 for the master branch, where I fixed some comments as well as thinkos in the test script. The ones on older branches aren't materially different, they just have tonnes of conflicts resolved. I'll get this pushed tomorrow morning. I have run it through

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-26 Thread Alvaro Herrera
On 2022-Jul-22, Kyotaro Horiguchi wrote: > At Fri, 22 Jul 2022 10:18:58 +0200, Alvaro Herrera > wrote in > > Which commits would we consider? > > > > 7170f2159fb2Allow "in place" tablespaces. > > f6f0db4d6240Fix pg_tablespace_location() with in-place tablespaces > > The second on

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-22 Thread Thomas Munro
On Fri, Jul 22, 2022 at 8:19 PM Alvaro Herrera wrote: > On 2022-Jul-22, Kyotaro Horiguchi wrote: > > At Thu, 21 Jul 2022 23:14:57 +1200, Thomas Munro > > wrote in > > > Would it help if we back-patched the allow_in_place_tablespaces stuff? > > > I'm not sure how hard/destabilising that would be

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-22 Thread Kyotaro Horiguchi
At Fri, 22 Jul 2022 10:18:58 +0200, Alvaro Herrera wrote in > OK, I'll wait for allow_in_place_tablespaces to be backpatched then. > > I would like to get this fix pushed before the next set of minors, so if > you won't have time for the backpatches early enough, maybe I can work > on getting i

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-22 Thread Alvaro Herrera
On 2022-Jul-22, Kyotaro Horiguchi wrote: > At Thu, 21 Jul 2022 23:14:57 +1200, Thomas Munro > wrote in > > Would it help if we back-patched the allow_in_place_tablespaces stuff? > > I'm not sure how hard/destabilising that would be, but I could take a > > look tomorrow. > > +1. Addiotional r

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-21 Thread Kyotaro Horiguchi
At Thu, 21 Jul 2022 13:25:05 +0200, Alvaro Herrera wrote in > On 2022-Jul-21, Alvaro Herrera wrote: > > > Yeah, I think that would reduce cruft. I'm not sure this is more > > against backpatching policy or less, compared to adding a separate > > GUC just for this bugfix. > > cruft: > > {

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-21 Thread Kyotaro Horiguchi
At Thu, 21 Jul 2022 23:14:57 +1200, Thomas Munro wrote in > On Thu, Jul 21, 2022 at 11:01 PM Alvaro Herrera > wrote: > > On 2022-Jul-20, Alvaro Herrera wrote: > > > I see the following alternatives: > > > > > > 1. not backpatch this fix to 14 and older > > > 2. use a different GUC; either allo

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-21 Thread Alvaro Herrera
On 2022-Jul-21, Alvaro Herrera wrote: > Yeah, I think that would reduce cruft. I'm not sure this is more > against backpatching policy or less, compared to adding a separate > GUC just for this bugfix. cruft: { {"allow_recovery_tablespaces", PG_POSTMASTER, WAL_RECOVERY,

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-21 Thread Alvaro Herrera
On 2022-Jul-21, Thomas Munro wrote: > On Thu, Jul 21, 2022 at 11:01 PM Alvaro Herrera > wrote: > > I've got no opinions on this. I don't like either 1 or 3, so I'm going > > to add and backpatch a new GUC allow_recovery_tablespaces as the > > override mechanism. > > > > If others disagree with

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-21 Thread Alvaro Herrera
On 2022-Jul-21, Thomas Munro wrote: > On Wed, Jul 20, 2022 at 10:51 PM Alvaro Herrera > wrote: > > v26 here. I spent some time fighting the readdir() stuff for > > Windows (so that get_dirent_type returns LNK for junction points) > > but couldn't make it to work and was unable to figure out why

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-21 Thread Thomas Munro
On Thu, Jul 21, 2022 at 11:01 PM Alvaro Herrera wrote: > On 2022-Jul-20, Alvaro Herrera wrote: > > I see the following alternatives: > > > > 1. not backpatch this fix to 14 and older > > 2. use a different GUC; either allow_invalid_pages as previously > >suggested, or create a new one just for

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-21 Thread Thomas Munro
On Wed, Jul 20, 2022 at 10:51 PM Alvaro Herrera wrote: > v26 here. I spent some time fighting the readdir() stuff for > Windows (so that get_dirent_type returns LNK for junction points) > but couldn't make it to work and was unable to figure out why. Was it because of this? https://www.postgres

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-21 Thread Alvaro Herrera
On 2022-Jul-20, Alvaro Herrera wrote: > I see the following alternatives: > > 1. not backpatch this fix to 14 and older > 2. use a different GUC; either allow_invalid_pages as previously >suggested, or create a new one just for this purpose > 3. not provide any overriding mechanism in version

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-20 Thread Alvaro Herrera
On 2022-Jul-20, Alvaro Herrera wrote: > On 2022-Jul-20, Alvaro Herrera wrote: > > > I also change the use of allow_invalid_pages to > > allow_in_place_tablespaces. We could add a > > separate GUC for this, but it seems overengineering. > > Oh, but allow_in_place_tablespaces doesn't exist in ver

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-20 Thread Alvaro Herrera
On 2022-Jul-20, Alvaro Herrera wrote: > I also change the use of allow_invalid_pages to > allow_in_place_tablespaces. We could add a > separate GUC for this, but it seems overengineering. Oh, but allow_in_place_tablespaces doesn't exist in versions 14 and older, so this strategy doesn't really w

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-20 Thread Alvaro Herrera
v26 here. I spent some time fighting the readdir() stuff for Windows (so that get_dirent_type returns LNK for junction points) but couldn't make it to work and was unable to figure out why. So I ended up doing what do_pg_backup_start is already doing: an #ifdef to call pgwin32_is_junction instead.

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-15 Thread Alvaro Herrera
On 2022-Jul-15, Alvaro Herrera wrote: > However, looking closer I noticed that on Windows we use our own > readdir() implementation, which AFAICT includes everything to handle > reparse points as symlinks correctly in get_dirent_type. Which means > that do_pg_start_backup is wasting its time with

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-15 Thread Alvaro Herrera
On 2022-Jul-15, Kyotaro Horiguchi wrote: > 0002: > +is_path_tslink(const char *path) > > What the "ts" of tslink stands for? If it stands for tablespace, the > function is not specific for table spaces. We already have > > + errmsg("could not stat file \"%s\"

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-15 Thread Alvaro Herrera
On 2022-Jul-15, Kyotaro Horiguchi wrote: > At Thu, 14 Jul 2022 23:47:40 +0200, Alvaro Herrera > wrote in > > Here's a couple of fixups. 0001 is the same as before. In 0002 I think > > Thanks! > > + if (!S_ISLNK(st.st_mode)) > + #else > + if (!pgwin32_is_junction(path

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-15 Thread Kyotaro Horiguchi
At Thu, 14 Jul 2022 23:47:40 +0200, Alvaro Herrera wrote in > Here's a couple of fixups. 0001 is the same as before. In 0002 I think Thanks! + if (!S_ISLNK(st.st_mode)) + #else + if (!pgwin32_is_junction(path)) + #endif + elog(ignore_invalid_

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-14 Thread Alvaro Herrera
Here's a couple of fixups. 0001 is the same as before. In 0002 I think CheckTablespaceDirectory ends up easier to read if we split out the test for validity of the link. Looking at that again, I think we don't need to piggyback on ignore_invalid_pages, which is already a stretch, so let's not --

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-07-13 Thread Alvaro Herrera
Not a review, just a preparatory rebase across some trivially conflicting changes. I also noticed that src/test/recovery/t/031_recovery_conflict.pl, which was added two days after v23 was sent, and which uses allow_in_place_tablespaces, bails out because of the checks introduced by this patch, so

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-04-05 Thread Kyotaro Horiguchi
At Tue, 05 Apr 2022 16:38:06 +0900 (JST), Kyotaro Horiguchi wrote in > At Tue, 05 Apr 2022 11:16:44 +0900 (JST), Kyotaro Horiguchi > wrote in > > So, I have the following points in my mind for now. > > > > - We create the directory "since we know it is just tentative state". > > > > - Then,

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-04-05 Thread Kyotaro Horiguchi
At Tue, 05 Apr 2022 16:38:06 +0900 (JST), Kyotaro Horiguchi wrote in > > However, while working on it, I found that I found that recovery faces > > missing tablespace directories *after* reaching consistency. I'm > > examining that further. > > Okay, it was my thinko. But I faced another obst

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-04-05 Thread Kyotaro Horiguchi
At Tue, 05 Apr 2022 11:16:44 +0900 (JST), Kyotaro Horiguchi wrote in > So, I have the following points in my mind for now. > > - We create the directory "since we know it is just tentative state". > > - Then, check that no directory in pg_tblspc when reaching consistency > when allow_in_plac

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-04-04 Thread Kyotaro Horiguchi
At Mon, 4 Apr 2022 21:14:27 +0530, Dilip Kumar wrote in > On Mon, Apr 4, 2022 at 2:25 PM Kyotaro Horiguchi > wrote: > > > > At Mon, 04 Apr 2022 17:29:48 +0900 (JST), Kyotaro Horiguchi > > wrote in > > > I haven't found how the patch caused creation of a relation file that > > > is to be remove

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-04-04 Thread Dilip Kumar
On Mon, Apr 4, 2022 at 2:25 PM Kyotaro Horiguchi wrote: > > At Mon, 04 Apr 2022 17:29:48 +0900 (JST), Kyotaro Horiguchi > wrote in > > I haven't found how the patch caused creation of a relation file that > > is to be removed soon. However, I find that v19 patch fails by maybe > > due to some c

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-04-04 Thread Kyotaro Horiguchi
At Mon, 04 Apr 2022 17:29:48 +0900 (JST), Kyotaro Horiguchi wrote in > I haven't found how the patch caused creation of a relation file that > is to be removed soon. However, I find that v19 patch fails by maybe > due to some change in Cluster.pm. It takes a bit more time to check > that.. I

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-04-04 Thread Kyotaro Horiguchi
At Fri, 1 Apr 2022 14:51:58 -0400, Robert Haas wrote in > On Fri, Apr 1, 2022 at 12:22 AM Kyotaro Horiguchi > wrote: > > By the way, may I ask how do we fix this? The existing recovery code > > already generates just-to-be-delete files in a real directory in > > pg_tblspc sometimes, and elsewis

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-04-01 Thread Robert Haas
On Fri, Apr 1, 2022 at 12:22 AM Kyotaro Horiguchi wrote: > By the way, may I ask how do we fix this? The existing recovery code > already generates just-to-be-delete files in a real directory in > pg_tblspc sometimes, and elsewise skip applying WAL records on > nonexistent heap pages. It is the

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-31 Thread Kyotaro Horiguchi
At Tue, 29 Mar 2022 09:31:42 -0400, Robert Haas wrote in > On Tue, Mar 29, 2022 at 9:28 AM Alvaro Herrera > wrote: > > OK, this is a bug that's been open for years. A fix can be committed > > after the feature freeze anyway. > > +1 By the way, may I ask how do we fix this? The existing re

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-29 Thread Robert Haas
On Tue, Mar 29, 2022 at 9:28 AM Alvaro Herrera wrote: > OK, this is a bug that's been open for years. A fix can be committed > after the feature freeze anyway. +1 -- Robert Haas EDB: http://www.enterprisedb.com

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-29 Thread Alvaro Herrera
On 2022-Mar-29, Robert Haas wrote: > I'm fine with that approach, but I'd like to ask that we proceed > expeditiously, because I have another patch that I want to commit that > touches this area. I can commit to helping with whatever we decide to > do here, but I don't want to keep that patch on i

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-29 Thread Robert Haas
On Tue, Mar 29, 2022 at 7:37 AM Alvaro Herrera wrote: > I think we should revert this patch and do it again using the other > approach: create a stub directory during recovery that can be deleted > later. I'm fine with that approach, but I'd like to ask that we proceed expeditiously, because I ha

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-29 Thread Alvaro Herrera
On 2022-Mar-29, Kyotaro Horiguchi wrote: > > That's going to result in a call to > > XLogReadBufferForRedoExtended() which will call > > XLogReadBufferExtended() which will do smgrcreate(smgr, forknum, true) > > which will in turn call TablespaceCreateDbspace() to fill in all the > > missing direc

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-28 Thread Kyotaro Horiguchi
At Mon, 28 Mar 2022 12:17:50 -0400, Robert Haas wrote in > On Mon, Mar 21, 2022 at 3:02 PM Alvaro Herrera > wrote: > > > 2. Why not instead change the code so that the operation can succeed, > > > by creating the prerequisite parent directories? Do we not have enough > > > information for that

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-28 Thread Michael Paquier
On Mon, Mar 28, 2022 at 02:34:44PM +1300, Thomas Munro wrote: > Just a thought: we could consider back-patching > allow_in_place_tablespaces, after a little while, if we're happy with > how that is working out, if it'd be useful for verifying bug fixes in > back branches. It's non-end-user-facing

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-28 Thread Kyotaro Horiguchi
At Mon, 28 Mar 2022 10:37:04 -0400, Robert Haas wrote in > On Fri, Mar 25, 2022 at 8:26 AM Alvaro Herrera > wrote: > > On 2022-Mar-21, Alvaro Herrera wrote: > > > I had a look at this latest version of the patch, and found some things > > > to tweak. Attached is v21 with three main changes fr

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-28 Thread Robert Haas
On Mon, Mar 21, 2022 at 3:02 PM Alvaro Herrera wrote: > > 2. Why not instead change the code so that the operation can succeed, > > by creating the prerequisite parent directories? Do we not have enough > > information for that? I'm not saying that we definitely should do it > > that way rather th

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-28 Thread Robert Haas
On Fri, Mar 25, 2022 at 8:26 AM Alvaro Herrera wrote: > On 2022-Mar-21, Alvaro Herrera wrote: > > I had a look at this latest version of the patch, and found some things > > to tweak. Attached is v21 with three main changes from Kyotaro's v20: > > Pushed this, backpatching to 14 and 13. It would

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-28 Thread Kyotaro Horiguchi
At Mon, 28 Mar 2022 10:01:05 +0900 (JST), Kyotaro Horiguchi wrote in > At Fri, 25 Mar 2022 13:26:05 +0100, Alvaro Herrera > wrote in > > Pushed this, backpatching to 14 and 13. It would have been good to > > backpatch further, but there's an (textually trivial) merge conflict > > related to

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-27 Thread Kyotaro Horiguchi
At Mon, 28 Mar 2022 14:34:44 +1300, Thomas Munro wrote in > On Mon, Mar 28, 2022 at 2:01 PM Kyotaro Horiguchi > wrote: > > At Fri, 25 Mar 2022 13:26:05 +0100, Alvaro Herrera > > wrote in > > > Pushed this, backpatching to 14 and 13. It would have been good to > > > backpatch further, but the

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-27 Thread Thomas Munro
On Mon, Mar 28, 2022 at 2:01 PM Kyotaro Horiguchi wrote: > At Fri, 25 Mar 2022 13:26:05 +0100, Alvaro Herrera > wrote in > > Pushed this, backpatching to 14 and 13. It would have been good to > > backpatch further, but there's an (textually trivial) merge conflict > > related to commit e6d80695

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-27 Thread Kyotaro Horiguchi
At Fri, 25 Mar 2022 13:26:05 +0100, Alvaro Herrera wrote in > On 2022-Mar-21, Alvaro Herrera wrote: > > > I had a look at this latest version of the patch, and found some things > > to tweak. Attached is v21 with three main changes from Kyotaro's v20: > > Pushed this, backpatching to 14 and 1

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-25 Thread Alvaro Herrera
On 2022-Mar-21, Alvaro Herrera wrote: > I had a look at this latest version of the patch, and found some things > to tweak. Attached is v21 with three main changes from Kyotaro's v20: Pushed this, backpatching to 14 and 13. It would have been good to backpatch further, but there's an (textually

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-21 Thread Alvaro Herrera
On 2022-Mar-14, Robert Haas wrote: > 2. Why not instead change the code so that the operation can succeed, > by creating the prerequisite parent directories? Do we not have enough > information for that? I'm not saying that we definitely should do it > that way rather than this way, but I think we

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-21 Thread Alvaro Herrera
I had a look at this latest version of the patch, and found some things to tweak. Attached is v21 with three main changes from Kyotaro's v20: 1. the XLogFlush is only done if consistent state has not been reached. As I understand, it's not needed in normal mode. (In any case, if we do call XLogF

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-21 Thread Alvaro Herrera
On 2022-Mar-04, Michael Paquier wrote: > d6d317d as solved the issue of tablespace paths across multiple nodes > with the new GUC called allow_in_place_tablespaces, and is getting > successfully used in the recovery tests as of 027_stream_regress.pl. OK, but that means that the test suite is now

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-14 Thread Kyotaro Horiguchi
At Mon, 14 Mar 2022 17:37:40 -0400, Robert Haas wrote in > On Mon, Mar 7, 2022 at 3:39 AM Kyotaro Horiguchi > wrote: > > So the new framework has been dropped in this version. > > The second test is removed as it is irrelevant to this bug. > > > > In this version the patch is a single file that

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-14 Thread Robert Haas
On Mon, Mar 7, 2022 at 3:39 AM Kyotaro Horiguchi wrote: > So the new framework has been dropped in this version. > The second test is removed as it is irrelevant to this bug. > > In this version the patch is a single file that contains the test. The status of this patch in the CommitFest was set

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-07 Thread Kyotaro Horiguchi
So the new framework has been dropped in this version. The second test is removed as it is irrelevant to this bug. In this version the patch is a single file that contains the test. regards. -- Kyotaro Horiguchi NTT Open Source Software Center >From 43bb3ba8900edd53a1feb0acb1a72bdc22bb1627 Mon

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-03 Thread Kyotaro Horiguchi
Thanks to look this! At Fri, 4 Mar 2022 13:51:12 +0900, Michael Paquier wrote i n > On Fri, Mar 04, 2022 at 09:10:48AM +0900, Kyotaro Horiguchi wrote: > > And same function contained a maybe-should-have-been-removed line > > which makes Windows build unhappy. > > > > This should make all platfo

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-03 Thread Michael Paquier
On Fri, Mar 04, 2022 at 09:10:48AM +0900, Kyotaro Horiguchi wrote: > And same function contained a maybe-should-have-been-removed line > which makes Windows build unhappy. > > This should make all platforms in the CI happy. d6d317d as solved the issue of tablespace paths across multiple nodes wit

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-03 Thread Kyotaro Horiguchi
At Wed, 02 Mar 2022 19:31:24 +0900 (JST), Kyotaro Horiguchi wrote in > A function added to Util.pm used perl2host, which has been removed > recently. And same function contained a maybe-should-have-been-removed line which makes Windows build unhappy. This should make all platforms in the CI ha

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-02 Thread Kyotaro Horiguchi
At Wed, 02 Mar 2022 16:59:09 +0900 (JST), Kyotaro Horiguchi wrote in > At Thu, 20 Jan 2022 17:19:04 +0900 (JST), Kyotaro Horiguchi > wrote in > > At Thu, 20 Jan 2022 15:07:22 +0900 (JST), Kyotaro Horiguchi > > wrote in > > CI now likes this version for all platforms. > > An xlog.c refacto

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-03-01 Thread Kyotaro Horiguchi
At Thu, 20 Jan 2022 17:19:04 +0900 (JST), Kyotaro Horiguchi wrote in > At Thu, 20 Jan 2022 15:07:22 +0900 (JST), Kyotaro Horiguchi > wrote in > CI now likes this version for all platforms. An xlog.c refactoring happend recently hit this. Just rebased on the change. regards. -- Kyotaro Hor

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-01-20 Thread Kyotaro Horiguchi
At Thu, 20 Jan 2022 15:07:22 +0900 (JST), Kyotaro Horiguchi wrote in > This version works for Unixen but still doesn't for Windows. I'm > searching for a fix for Windows. And this version works for Windows. Maybe I've took a wrong version to post. dir_readlink manipulated target file (junction

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-01-19 Thread Kyotaro Horiguchi
At Sun, 16 Jan 2022 12:43:03 +0800, Julien Rouhaud wrote in > Other than that, I see that the TAP tests are failing on all the environment, > due to Perl errors. For instance: Perl seems to have changed its behavior for undef hash. It is said that "if (%undef_hash)" is false but actually it i

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-01-19 Thread Kyotaro Horiguchi
At Mon, 17 Jan 2022 17:24:43 +0900 (JST), Kyotaro Horiguchi wrote in > At Sun, 16 Jan 2022 12:43:03 +0800, Julien Rouhaud wrote > in > > I'm not very familiar with windows, but maybe using strawberry perl instead > > ([1]) would fix your problem? I think it's also quite popular and is > > c

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-01-17 Thread Kyotaro Horiguchi
At Sun, 16 Jan 2022 12:43:03 +0800, Julien Rouhaud wrote in > Hi, > > On Fri, Dec 24, 2021 at 07:21:59PM +0900, Kyotaro Horiguchi wrote: > > Just a complaint.. > > > > At Fri, 12 Nov 2021 16:43:27 +0900 (JST), Kyotaro Horiguchi > > wrote in > > > "" on Japanese (CP-932) environment. I didn'

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-01-15 Thread Julien Rouhaud
Hi, On Fri, Dec 24, 2021 at 07:21:59PM +0900, Kyotaro Horiguchi wrote: > Just a complaint.. > > At Fri, 12 Nov 2021 16:43:27 +0900 (JST), Kyotaro Horiguchi > wrote in > > "" on Japanese (CP-932) environment. I didn't actually > > tested it on Windows and msys environment ...yet. > > Active pe

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2021-12-24 Thread Kyotaro Horiguchi
Just a complaint.. At Fri, 12 Nov 2021 16:43:27 +0900 (JST), Kyotaro Horiguchi wrote in > "" on Japanese (CP-932) environment. I didn't actually > tested it on Windows and msys environment ...yet. Active perl cannot be installed because of (perhaps) a powershell version issue... Annoying.. ht

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2021-11-11 Thread Kyotaro Horiguchi
At Thu, 11 Nov 2021 11:13:52 +0900 (JST), Kyotaro Horiguchi wrote in > At Wed, 10 Nov 2021 09:14:30 -0300, Alvaro Herrera > wrote in > > Can you use PostgreSQL::Test::Utils::tempdir_short() for those > > tablespaces? > > Thanks for the suggestion! > > It works for a live cluster. But doesn'

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2021-11-10 Thread Kyotaro Horiguchi
At Wed, 10 Nov 2021 09:14:30 -0300, Alvaro Herrera wrote in > Can you use PostgreSQL::Test::Utils::tempdir_short() for those > tablespaces? Thanks for the suggestion! It works for a live cluster. But doesn't work for backups, since I find no way to relate a tablespace directory with a backup d

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2021-11-10 Thread Alvaro Herrera
On 2021-Nov-10, Kyotaro Horiguchi wrote: > I bumped into the good-old 100-byte limit of the (v7?) tar format on > which pg_basebackup is depending. It is unlikely in the real world but > I think it is quite common in developping environment. The tablespace > directory path in my dev environment w

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2021-11-10 Thread Kyotaro Horiguchi
At Tue, 09 Nov 2021 17:05:49 +0900 (JST), Kyotaro Horiguchi wrote in > At Tue, 9 Nov 2021 12:51:15 +0900, Michael Paquier > wrote in > > Look at Utils.pm where we have dir_symlink, then. symlink() does not > > work on WIN32, so we have a wrapper that uses junction points. FWIW, > > I don't

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2021-11-09 Thread Kyotaro Horiguchi
At Tue, 9 Nov 2021 12:51:15 +0900, Michael Paquier wrote in > On Mon, Nov 08, 2021 at 05:55:16PM +0900, Kyotaro Horiguchi wrote: > > I have quickly looked at the patch set. > > > 0001: (I don't remember about this, though) I don't see how to make it > > work on Windows. Anyway the next step w

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2021-11-08 Thread Michael Paquier
On Mon, Nov 08, 2021 at 05:55:16PM +0900, Kyotaro Horiguchi wrote: I have quickly looked at the patch set. > 0001: (I don't remember about this, though) I don't see how to make it > work on Windows. Anyway the next step would be to write comments. Look at Utils.pm where we have dir_symlink, the

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2021-11-08 Thread Kyotaro Horiguchi
At Thu, 4 Nov 2021 13:34:33 +0100, Daniel Gustafsson wrote in > This patch again fails to apply (seemingly from the Perl namespace work on the > testcode), and needs a few updates as per the above review. Rebased the latest patch removing some of the chages. 0001: (I don't remember about this,

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2021-11-04 Thread Daniel Gustafsson
> On 24 Sep 2021, at 20:14, Tom Lane wrote: > > Robert Haas writes: >> The commit message for 0001 is not clear enough for me to understand >> what problem it's supposed to be fixing. The code comments aren't >> really either. They make it sound like there's some problem with >> copying symlinks

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2021-09-24 Thread Tom Lane
Robert Haas writes: > The commit message for 0001 is not clear enough for me to understand > what problem it's supposed to be fixing. The code comments aren't > really either. They make it sound like there's some problem with > copying symlinks but mostly they just talk about callbacks, which > do

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2021-08-11 Thread Robert Haas
On Wed, Aug 11, 2021 at 3:59 AM Paul Guo wrote: > Thanks for reviewing. Let me explain a bit. The patch series includes > four patches. > > 0001 and 0002 are test changes for the fix (0003). >- 0001 is the test framework change that's needed by 0002. >- 0002 is the test for the code fix (0

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2021-08-11 Thread Paul Guo
On Wed, Aug 11, 2021 at 4:56 AM Robert Haas wrote: > > On Thu, Aug 5, 2021 at 6:20 AM Paul Guo wrote: > > Rebased. > > The commit message for 0001 is not clear enough for me to understand > what problem it's supposed to be fixing. The code comments aren't > really either. They make it sound like

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2021-08-10 Thread Robert Haas
On Thu, Aug 5, 2021 at 6:20 AM Paul Guo wrote: > Rebased. The commit message for 0001 is not clear enough for me to understand what problem it's supposed to be fixing. The code comments aren't really either. They make it sound like there's some problem with copying symlinks but mostly they just t

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2021-08-05 Thread Paul Guo
Rebased. v12-0002-Tests-to-replay-create-database-operation-on-sta.patch Description: v12-0002-Tests-to-replay-create-database-operation-on-sta.patch v12-0001-Support-node-initialization-from-backup-with-tab.patch Description: v12-0001-Support-node-initialization-from-backup-with-tab.patch

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2021-07-09 Thread Ibrar Ahmed
On Tue, Mar 30, 2021 at 12:12 PM Paul Guo wrote: > On 2021/3/27, 10:23 PM, "Alvaro Herrera" wrote: > > >Hmm, can you post a rebased set, where the points under discussion > > are marked in XXX comments explaining what the issue is? This thread > is > >long and old ago that it's pretty

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2021-03-30 Thread Paul Guo
On 2021/3/27, 10:23 PM, "Alvaro Herrera" wrote: >Hmm, can you post a rebased set, where the points under discussion > are marked in XXX comments explaining what the issue is? This thread is >long and old ago that it's pretty hard to navigate the whole thing in >order to find out ex

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2021-03-27 Thread Alvaro Herrera
On 2021-Jan-27, Paul Guo wrote: > Here is a git diff against the previous patch. I’ll send out the new > rebased patches after the consensus is reached. Hmm, can you post a rebased set, where the points under discussion are marked in XXX comments explaining what the issue is? This thread is long

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2021-01-27 Thread Paul Guo
Thanks for the review, please see the replies below. > On Jan 5, 2021, at 9:07 AM, Kyotaro Horiguchi wrote: > > At Wed, 8 Jul 2020 12:56:44 +, Paul Guo wrote in >> On 2020/01/15 19:18, Paul Guo wrote: >> I further fixed the last test failure (due to a small bug in the test, not >> in code

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2021-01-04 Thread Kyotaro Horiguchi
At Wed, 8 Jul 2020 12:56:44 +, Paul Guo wrote in > On 2020/01/15 19:18, Paul Guo wrote: > I further fixed the last test failure (due to a small bug in the test, not in > code). Attached are the new patch series. Let's see the CI pipeline result. > > Thanks for updating the patches! > > I s

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2020-07-08 Thread Paul Guo
Looks like my previous reply was held for moderation (maybe due to my new email address). I configured my pg account today using the new email address. I guess this email would be held for moderation. I’m now replying my previous reply email and attaching the new patch series. On Jul 6, 2020,

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2020-07-07 Thread Daniel Gustafsson
> On 25 Mar 2020, at 06:52, Fujii Masao wrote: > > On 2020/01/28 0:24, Fujii Masao wrote: >> On 2020/01/15 19:18, Paul Guo wrote: >>> I further fixed the last test failure (due to a small bug in the test, not >>> in code). Attached are the new patch series. Let's see the CI pipeline >>> result.

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2020-03-24 Thread Fujii Masao
On 2020/01/28 0:24, Fujii Masao wrote: On 2020/01/15 19:18, Paul Guo wrote: I further fixed the last test failure (due to a small bug in the test, not in code). Attached are the new patch series. Let's see the CI pipeline result. Thanks for updating the patches! I started reading the 00

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2020-01-27 Thread Fujii Masao
On 2020/01/15 19:18, Paul Guo wrote: I further fixed the last test failure (due to a small bug in the test, not in code). Attached are the new patch series. Let's see the CI pipeline result. Thanks for updating the patches! I started reading the 0003 patch. The approach that the 0003 patc

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2020-01-15 Thread Paul Guo
I further fixed the last test failure (due to a small bug in the test, not in code). Attached are the new patch series. Let's see the CI pipeline result. v9-0001-Support-node-initialization-from-backup-with-tabl.patch Description: Binary data v9-0003-Fix-replay-of-create-database-records-on-sta

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2020-01-13 Thread Paul Guo
On Fri, Jan 10, 2020 at 9:43 PM Alvaro Herrera wrote: > On 2020-Jan-09, Alvaro Herrera wrote: > > > I looked at this a little while and was bothered by the perl changes; it > > seems out of place to have RecursiveCopy be thinking about tablespaces, > > which is way out of its league. So I rewrot

  1   2   >