Re: pg_upgrade should truncate/remove its logs before running

2022-02-07 Thread Julien Rouhaud
On Mon, Feb 07, 2022 at 11:00:22AM -0500, Andrew Dunstan wrote: > > On 2/6/22 19:39, Tom Lane wrote: > > > > I note, though, that there's still not been any email to the buildfarm > > owners list about this update. > > The announcement was held up in list moderation for 20 hours or so. I've

Re: pg_upgrade should truncate/remove its logs before running

2022-02-07 Thread Andrew Dunstan
On 2/6/22 19:39, Tom Lane wrote: > Michael Paquier writes: >> On Sun, Feb 06, 2022 at 08:32:59AM -0500, Andrew Dunstan wrote: >>> But the commit really shouldn't have happened until we know that most >>> buildfarm owners have installed it. It should have waited wait not just >>> for the release

Re: pg_upgrade should truncate/remove its logs before running

2022-02-06 Thread Andrew Dunstan
> On Feb 6, 2022, at 7:39 PM, Tom Lane wrote: > > Michael Paquier writes: >>> On Sun, Feb 06, 2022 at 08:32:59AM -0500, Andrew Dunstan wrote: >>> But the commit really shouldn't have happened until we know that most >>> buildfarm owners have installed it. It should have waited wait not just

Re: pg_upgrade should truncate/remove its logs before running

2022-02-06 Thread Tom Lane
Michael Paquier writes: > On Sun, Feb 06, 2022 at 08:32:59AM -0500, Andrew Dunstan wrote: >> But the commit really shouldn't have happened until we know that most >> buildfarm owners have installed it. It should have waited wait not just >> for the release but for widespread deployment. Otherwise

Re: pg_upgrade should truncate/remove its logs before running

2022-02-06 Thread Michael Paquier
On Sun, Feb 06, 2022 at 08:32:59AM -0500, Andrew Dunstan wrote: > *sigh* Sometimes I have a mind like a sieve. I prepped the release a few > days ago and meant to come back the next morning and send out emails > announcing it, as well as rolling it out to my animals, and got diverted > so that

Re: pg_upgrade should truncate/remove its logs before running

2022-02-06 Thread Andrew Dunstan
On 2/6/22 02:17, Tom Lane wrote: > Michael Paquier writes: >> On Sun, Feb 06, 2022 at 01:58:21AM -0500, Tom Lane wrote: >>> As already mentioned, there's been no notice to buildfarm owners ... >>> so has Andrew actually made a release? >> There has been one as of 8 days ago: >>

Re: pg_upgrade should truncate/remove its logs before running

2022-02-05 Thread Tom Lane
Michael Paquier writes: > On Sun, Feb 06, 2022 at 01:58:21AM -0500, Tom Lane wrote: >> As already mentioned, there's been no notice to buildfarm owners ... >> so has Andrew actually made a release? > There has been one as of 8 days ago: > https://github.com/PGBuildFarm/client-code/releases [

Re: pg_upgrade should truncate/remove its logs before running

2022-02-05 Thread Michael Paquier
On Sun, Feb 06, 2022 at 01:58:21AM -0500, Tom Lane wrote: > As already mentioned, there's been no notice to buildfarm owners ... > so has Andrew actually made a release? There has been one as of 8 days ago: https://github.com/PGBuildFarm/client-code/releases And I have just looked at that as

Re: pg_upgrade should truncate/remove its logs before running

2022-02-05 Thread Julien Rouhaud
On Sun, Feb 06, 2022 at 01:58:21AM -0500, Tom Lane wrote: > Michael Paquier writes: > > So, it took me some time to get back to this thread, and looked at it > > for the last couple of days... The buildfarm client v14 has been > > released on the 29th of January, which means that we are good to

Re: pg_upgrade should truncate/remove its logs before running

2022-02-05 Thread Tom Lane
Michael Paquier writes: > So, it took me some time to get back to this thread, and looked at it > for the last couple of days... The buildfarm client v14 has been > released on the 29th of January, which means that we are good to go. As already mentioned, there's been no notice to buildfarm

Re: pg_upgrade should truncate/remove its logs before running

2022-02-05 Thread Michael Paquier
On Sun, Feb 06, 2022 at 02:03:44PM +0800, Julien Rouhaud wrote: > I didn't follow that thread closely, but if having the latest buildfarm client > version installed is a hard requirement this will likely be a problem. First, > there was no email to warn buildfarm owners that a new version is

Re: pg_upgrade should truncate/remove its logs before running

2022-02-05 Thread Julien Rouhaud
Hi, On Sun, Feb 06, 2022 at 01:36:07PM +0900, Michael Paquier wrote: > > The buildfarm client v14 has been > released on the 29th of January, which means that we are good to go. I didn't follow that thread closely, but if having the latest buildfarm client version installed is a hard

Re: pg_upgrade should truncate/remove its logs before running

2022-02-05 Thread Michael Paquier
On Sat, Jan 29, 2022 at 09:53:25AM +0900, Michael Paquier wrote: > On Fri, Jan 28, 2022 at 06:27:29PM -0500, Andrew Dunstan wrote: >> I have committed this. But it will take time to get every buildfarm own >> to upgrade. > > Thanks for that. So, it took me some time to get back to this thread,

Re: pg_upgrade should truncate/remove its logs before running

2022-01-28 Thread Michael Paquier
On Fri, Jan 28, 2022 at 06:27:29PM -0500, Andrew Dunstan wrote: > I have committed this. But it will take time to get every buildfarm own > to upgrade. Thanks for that. > I will try to make a new release ASAP. And thanks for that, as well. -- Michael signature.asc Description: PGP signature

Re: pg_upgrade should truncate/remove its logs before running

2022-01-28 Thread Andrew Dunstan
On 1/28/22 08:42, Michael Paquier wrote: > On Wed, Jan 26, 2022 at 11:00:28AM +0900, Michael Paquier wrote: >> Bleh. This would point to the old data directory, so this needs to be >> "$self->{pgsql}/src/bin/pg_upgrade/tmp_check/data/pg_upgrade_output.d/log/*.log" >> to point to the upgraded

Re: pg_upgrade should truncate/remove its logs before running

2022-01-28 Thread Michael Paquier
On Wed, Jan 26, 2022 at 11:00:28AM +0900, Michael Paquier wrote: > Bleh. This would point to the old data directory, so this needs to be > "$self->{pgsql}/src/bin/pg_upgrade/tmp_check/data/pg_upgrade_output.d/log/*.log" > to point to the upgraded cluster. Please note that I have sent a patch to

Re: pg_upgrade should truncate/remove its logs before running

2022-01-25 Thread Michael Paquier
On Wed, Jan 26, 2022 at 09:44:48AM +0900, Michael Paquier wrote: > Yes, this is going to need an adjustment of @logfiles in > TestUpgrade.pm, with the addition of > "$tmp_data_dir/pg_update_output.d/log/*.log" to be consistent with the > data fetched for the tests of older branches. Bleh. This

Re: pg_upgrade should truncate/remove its logs before running

2022-01-25 Thread Michael Paquier
On Tue, Jan 25, 2022 at 10:45:29AM -0600, Justin Pryzby wrote: > Andrew: you wanted to accommodate any change on the build client, right ? Yes, this is going to need an adjustment of @logfiles in TestUpgrade.pm, with the addition of "$tmp_data_dir/pg_update_output.d/log/*.log" to be consistent

Re: pg_upgrade should truncate/remove its logs before running

2022-01-25 Thread Justin Pryzby
On Mon, Jan 24, 2022 at 10:59:40AM +0900, Michael Paquier wrote: > On Thu, Jan 20, 2022 at 07:51:37PM +0900, Michael Paquier wrote: > > Neat idea. That would work fine for my case. So I am fine to stick > > with this suggestion. > > I have been looking at this idea, and the result is quite

Re: pg_upgrade should truncate/remove its logs before running

2022-01-24 Thread Michael Paquier
On Mon, Jan 24, 2022 at 02:44:21PM -0500, Bruce Momjian wrote: > OK, thanks. There are really two cleanups --- first, the "log" > directory, and second deletion of the old cluster by running > delete_old_cluster.sh. Yes, this is the same thing as what's done on HEAD with a two-step cleanup,

Re: pg_upgrade should truncate/remove its logs before running

2022-01-24 Thread Bruce Momjian
On Mon, Jan 24, 2022 at 11:41:17AM -0600, Justin Pryzby wrote: > On Mon, Jan 24, 2022 at 12:39:30PM -0500, Bruce Momjian wrote: > > On Mon, Jan 24, 2022 at 10:59:40AM +0900, Michael Paquier wrote: > > > On Thu, Jan 20, 2022 at 07:51:37PM +0900, Michael Paquier wrote: > > > > Neat idea. That would

Re: pg_upgrade should truncate/remove its logs before running

2022-01-24 Thread Bruce Momjian
On Mon, Jan 24, 2022 at 10:59:40AM +0900, Michael Paquier wrote: > On Thu, Jan 20, 2022 at 07:51:37PM +0900, Michael Paquier wrote: > > Neat idea. That would work fine for my case. So I am fine to stick > > with this suggestion. > > I have been looking at this idea, and the result is quite

Re: pg_upgrade should truncate/remove its logs before running

2022-01-24 Thread Justin Pryzby
On Mon, Jan 24, 2022 at 12:39:30PM -0500, Bruce Momjian wrote: > On Mon, Jan 24, 2022 at 10:59:40AM +0900, Michael Paquier wrote: > > On Thu, Jan 20, 2022 at 07:51:37PM +0900, Michael Paquier wrote: > > > Neat idea. That would work fine for my case. So I am fine to stick > > > with this

Re: pg_upgrade should truncate/remove its logs before running

2022-01-23 Thread Michael Paquier
On Thu, Jan 20, 2022 at 07:51:37PM +0900, Michael Paquier wrote: > Neat idea. That would work fine for my case. So I am fine to stick > with this suggestion. I have been looking at this idea, and the result is quite nice, being simpler than anything that has been proposed on this thread yet.

Re: pg_upgrade should truncate/remove its logs before running

2022-01-20 Thread Michael Paquier
On Thu, Jan 20, 2022 at 10:31:15AM +0100, Peter Eisentraut wrote: > I'm afraid that is too easily confused with the target directory. Generally, > a tool processes data from input to output or from source to target or > something like that, whereas a log is more clearly something separate from >

Re: pg_upgrade should truncate/remove its logs before running

2022-01-20 Thread Peter Eisentraut
On 19.01.22 09:13, Michael Paquier wrote: - Renaming of the option from --logdir to --outputdir, as this does not include only logs. That matches also better with default value assigned in previous patches, aka pg_upgrade_output.d. I'm afraid that is too easily confused with the target

Re: pg_upgrade should truncate/remove its logs before running

2022-01-19 Thread Michael Paquier
On Wed, Jan 19, 2022 at 09:59:14PM -0600, Justin Pryzby wrote: > We require that the dir not exist, by testing if (mkdir()). > So it's okay if someone specifies ../whatever or $CWD. What I am scared of here is the use of rmtree() if we allow something like that. So we should either keep the

Re: pg_upgrade should truncate/remove its logs before running

2022-01-19 Thread Justin Pryzby
On Thu, Jan 20, 2022 at 12:01:29PM +0900, Michael Paquier wrote: > On Wed, Jan 19, 2022 at 06:05:40PM -0600, Justin Pryzby wrote: > > > I'm not sure these restrictions are needed ? > > This could lead to issues with rmtree() if we are not careful enough, > no? We'd had our deal of argument

Re: pg_upgrade should truncate/remove its logs before running

2022-01-19 Thread Michael Paquier
On Wed, Jan 19, 2022 at 06:05:40PM -0600, Justin Pryzby wrote: > I still don't know if it even needs to be configurable. I want this to be configurable to ease the switch of the pg_upgrade to TAP, moving the logs into a deterministic temporary location proper to each run. This makes reporting

Re: pg_upgrade should truncate/remove its logs before running

2022-01-19 Thread Justin Pryzby
On Wed, Jan 19, 2022 at 05:13:18PM +0900, Michael Paquier wrote: > On Tue, Jan 11, 2022 at 10:08:13PM -0600, Justin Pryzby wrote: > > I asked about that before. Right now, it'll exit(1) when mkdir fails. > > > > I had written a patch to allow "." by skipping mkdir (or allowing it to > > fail if

Re: pg_upgrade should truncate/remove its logs before running

2022-01-19 Thread Michael Paquier
On Wed, Jan 19, 2022 at 05:13:18PM +0900, Michael Paquier wrote: > I have noticed a couple of incorrect things in the docs, and some > other things. It is a bit late here, so I may have missed a couple of > things but I'll look at this stuff once again in a couple of days. And the docs failed to

Re: pg_upgrade should truncate/remove its logs before running

2022-01-19 Thread Michael Paquier
On Tue, Jan 11, 2022 at 10:08:13PM -0600, Justin Pryzby wrote: > I asked about that before. Right now, it'll exit(1) when mkdir fails. > > I had written a patch to allow "." by skipping mkdir (or allowing it to fail > if > errno == EEXIST), but it seems like an awfully bad idea to try to make

Re: pg_upgrade should truncate/remove its logs before running

2022-01-11 Thread Justin Pryzby
On Wed, Jan 12, 2022 at 12:59:54PM +0900, Michael Paquier wrote: > On Tue, Jan 11, 2022 at 02:03:07PM -0600, Justin Pryzby wrote: > > There's no reason not to. We created the dir, and the user didn't specify > > to > > preserve it. It'd be their fault if they put something valuable there after

Re: pg_upgrade should truncate/remove its logs before running

2022-01-11 Thread Michael Paquier
On Tue, Jan 11, 2022 at 02:03:07PM -0600, Justin Pryzby wrote: > I added mkdir() before the other stuff that messes with logfiles, because it > needs to happen before that. > > Are you suggesting to change the pre-existing behavior of when logfiles are > created, like 0002 ? Yes, something like

Re: pg_upgrade should truncate/remove its logs before running

2022-01-11 Thread Justin Pryzby
On Tue, Jan 11, 2022 at 04:41:58PM +0900, Michael Paquier wrote: > On Sat, Jan 08, 2022 at 12:48:57PM -0600, Justin Pryzby wrote: > > I fixed it by calling get_restricted_token() before parseCommandLine(). > > There's precedent for that in pg_regress (but the 3 other callers do it > >

Re: pg_upgrade should truncate/remove its logs before running

2022-01-10 Thread Michael Paquier
On Sat, Jan 08, 2022 at 12:48:57PM -0600, Justin Pryzby wrote: > I fixed it by calling get_restricted_token() before parseCommandLine(). > There's precedent for that in pg_regress (but the 3 other callers do it > differently). > > It seems more ideal to always call get_restricted_token sooner than

Re: pg_upgrade should truncate/remove its logs before running

2022-01-08 Thread Justin Pryzby
The cfbot was failing under windows: | [22:07:02.159] could not create directory "pg_upgrade_output.d": File exists It's because parseCommandLine() was called before get_restricted_token(), which re-executes the process, and runs parseCommandLine again. parseCommandLine already does stuff like

Re: pg_upgrade should truncate/remove its logs before running

2021-12-22 Thread Justin Pryzby
On Wed, Dec 22, 2021 at 09:52:26AM -0500, Tom Lane wrote: > Michael Paquier writes: > > On Mon, Dec 20, 2021 at 09:39:26PM -0600, Justin Pryzby wrote: > >> Are you suggesting to remove these ? > >> -/pg_upgrade_internal.log > >> -/loadable_libraries.txt > > > Yep, it looks so as these are part

Re: pg_upgrade should truncate/remove its logs before running

2021-12-22 Thread Tom Lane
Michael Paquier writes: > On Mon, Dec 20, 2021 at 09:39:26PM -0600, Justin Pryzby wrote: >> Are you suggesting to remove these ? >> -/pg_upgrade_internal.log >> -/loadable_libraries.txt > Yep, it looks so as these are part of the logs, the second one being a > failure state. >>

Re: pg_upgrade should truncate/remove its logs before running

2021-12-21 Thread Michael Paquier
On Mon, Dec 20, 2021 at 09:39:26PM -0600, Justin Pryzby wrote: > On Mon, Dec 20, 2021 at 08:21:51PM +0900, Michael Paquier wrote: >> we could choose something simpler for the default, like >> "pg_upgrade_log". I don't have a good history in naming new things, >> though :) > > I specifically

Re: pg_upgrade should truncate/remove its logs before running

2021-12-20 Thread Justin Pryzby
On Mon, Dec 20, 2021 at 08:21:51PM +0900, Michael Paquier wrote: > On Fri, Dec 17, 2021 at 11:21:13AM -0600, Justin Pryzby wrote: > + log_opts.basedir = "pg_upgrade_log.d"; > we could choose something simpler for the default, like > "pg_upgrade_log". I don't have a good history in

Re: pg_upgrade should truncate/remove its logs before running

2021-12-20 Thread Michael Paquier
On Fri, Dec 17, 2021 at 11:21:13AM -0600, Justin Pryzby wrote: > I put this together in the simplest way, prefixing all the filenames with the > configured path.. Well, why not. > Another options is to chdir() into the given path. But, pg_upgrade takes (and > requires) a bunch of other paths,

Re: pg_upgrade should truncate/remove its logs before running

2021-12-17 Thread Justin Pryzby
On Thu, Dec 16, 2021 at 12:23:08PM +0100, Daniel Gustafsson wrote: > > On 16 Dec 2021, at 12:11, Peter Eisentraut > > wrote: > > > Could we make it write just one log file? Is having multiple log files > > better? > > Having individual .txt files from checks with additional > information >

Re: pg_upgrade should truncate/remove its logs before running

2021-12-16 Thread Daniel Gustafsson
> On 16 Dec 2021, at 12:11, Peter Eisentraut > wrote: > Could we make it write just one log file? Is having multiple log files > better? Having individual .txt files from checks with additional information on how to handle the error are quite convenient when writing wrappers around

Re: pg_upgrade should truncate/remove its logs before running

2021-12-16 Thread Peter Eisentraut
On 16.12.21 02:39, Michael Paquier wrote: On Wed, Dec 15, 2021 at 04:13:10PM -0600, Justin Pryzby wrote: On Wed, Dec 15, 2021 at 05:04:54PM -0500, Andrew Dunstan wrote: The directory name needs to be predictable somehow, or maybe optionally set as a parameter. Having just a timestamped

Re: pg_upgrade should truncate/remove its logs before running

2021-12-15 Thread Michael Paquier
On Wed, Dec 15, 2021 at 04:13:10PM -0600, Justin Pryzby wrote: > On Wed, Dec 15, 2021 at 05:04:54PM -0500, Andrew Dunstan wrote: >> The directory name needs to be predictable somehow, or maybe optionally >> set as a parameter. Having just a timestamped directory name would make >> life annoying

Re: pg_upgrade should truncate/remove its logs before running

2021-12-15 Thread Justin Pryzby
On Wed, Dec 15, 2021 at 05:04:54PM -0500, Andrew Dunstan wrote: > On 12/15/21 16:23, Bruce Momjian wrote: > > On Wed, Dec 15, 2021 at 04:17:23PM -0500, Tom Lane wrote: > >> Bruce Momjian writes: > >>> On Sat, Dec 11, 2021 at 08:50:17PM -0600, Justin Pryzby wrote: > If pg_upgrade fails and is

Re: pg_upgrade should truncate/remove its logs before running

2021-12-15 Thread Andrew Dunstan
On 12/15/21 16:23, Bruce Momjian wrote: > On Wed, Dec 15, 2021 at 04:17:23PM -0500, Tom Lane wrote: >> Bruce Momjian writes: >>> On Sat, Dec 11, 2021 at 08:50:17PM -0600, Justin Pryzby wrote: If pg_upgrade fails and is re-run, it appends to its logfiles, which is confusing since, if

Re: pg_upgrade should truncate/remove its logs before running

2021-12-15 Thread Bruce Momjian
On Wed, Dec 15, 2021 at 04:17:23PM -0500, Tom Lane wrote: > Bruce Momjian writes: > > On Sat, Dec 11, 2021 at 08:50:17PM -0600, Justin Pryzby wrote: > >> If pg_upgrade fails and is re-run, it appends to its logfiles, which is > >> confusing since, if it fails again, it then looks like the

Re: pg_upgrade should truncate/remove its logs before running

2021-12-15 Thread Tom Lane
Bruce Momjian writes: > On Sat, Dec 11, 2021 at 08:50:17PM -0600, Justin Pryzby wrote: >> If pg_upgrade fails and is re-run, it appends to its logfiles, which is >> confusing since, if it fails again, it then looks like the original error >> recurred and wasn't fixed. The "append" behavior dates

Re: pg_upgrade should truncate/remove its logs before running

2021-12-15 Thread Justin Pryzby
On Wed, Dec 15, 2021 at 04:09:16PM -0500, Bruce Momjian wrote: > On Sat, Dec 11, 2021 at 08:50:17PM -0600, Justin Pryzby wrote: > > I have seen this numerous times but had not dug into it, until now. > > > > If pg_upgrade fails and is re-run, it appends to its logfiles, which is > > confusing

Re: pg_upgrade should truncate/remove its logs before running

2021-12-15 Thread Bruce Momjian
On Sat, Dec 11, 2021 at 08:50:17PM -0600, Justin Pryzby wrote: > I have seen this numerous times but had not dug into it, until now. > > If pg_upgrade fails and is re-run, it appends to its logfiles, which is > confusing since, if it fails again, it then looks like the original error > recurred

pg_upgrade should truncate/remove its logs before running

2021-12-11 Thread Justin Pryzby
I have seen this numerous times but had not dug into it, until now. If pg_upgrade fails and is re-run, it appends to its logfiles, which is confusing since, if it fails again, it then looks like the original error recurred and wasn't fixed. The "append" behavior dates back to 717f6d608. I think