Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-07-08 Thread Nathan Bossart
Committed. -- nathan

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-19 Thread Michael Paquier
On Wed, Jun 19, 2024 at 08:37:17AM -0500, Nathan Bossart wrote: > My understanding is that you basically have to restart the upgrade from > scratch if that happens. I suppose there could be a problem if you try to > use the half-upgraded cluster after a crash, but I imagine you have a good >

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-19 Thread Matthias van de Meent
On Wed, 19 Jun 2024 at 15:17, Robert Haas wrote: > > On Fri, Jun 14, 2024 at 5:29 PM Nathan Bossart > wrote: > > On Wed, Jun 12, 2024 at 09:41:01AM -0500, Nathan Bossart wrote: > > > Actually, I think you are right that we need a manual checkpoint, except I > > > think we need it to be after

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-19 Thread Nathan Bossart
On Wed, Jun 19, 2024 at 09:17:00AM -0400, Robert Haas wrote: > OK, I have a (probably) stupid question. The comment says: > > + * In binary upgrade mode, we can skip this checkpoint because neither of > + * these problems applies: we don't ever replay the WAL generated during > + * pg_upgrade,

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-19 Thread Robert Haas
On Fri, Jun 14, 2024 at 5:29 PM Nathan Bossart wrote: > On Wed, Jun 12, 2024 at 09:41:01AM -0500, Nathan Bossart wrote: > > Actually, I think you are right that we need a manual checkpoint, except I > > think we need it to be after prepare_new_globals(). set_frozenxids() > > connects to each

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-19 Thread Matthias van de Meent
On Fri, 14 Jun 2024 at 23:29, Nathan Bossart wrote: > > On Wed, Jun 12, 2024 at 09:41:01AM -0500, Nathan Bossart wrote: > > Actually, I think you are right that we need a manual checkpoint, except I > > think we need it to be after prepare_new_globals(). set_frozenxids() > > connects to each

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-14 Thread Nathan Bossart
00 Subject: [PATCH v3 1/1] Use CREATE DATABASE ... STRATEGY = FILE_COPY in pg_upgrade. While the strategy is considered very costly due to expensive checkpoints when the target system has configured large shared_buffers, the issues that those checkpoints protect against don't apply in a

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-12 Thread Nathan Bossart
On Tue, Jun 11, 2024 at 10:39:51AM +0200, Matthias van de Meent wrote: > On Tue, 11 Jun 2024 at 04:01, Nathan Bossart wrote: >> It looks like pg_dump always uses template0, so AFAICT we don't even need >> the suggested manual checkpoint after restoring template1. > > Thanks for reminding me. It

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-11 Thread Nathan Bossart
On Tue, Jun 11, 2024 at 10:39:51AM +0200, Matthias van de Meent wrote: > On Tue, 11 Jun 2024 at 04:01, Nathan Bossart wrote: >> I did a handful of benchmarks on an r5.24xlarge that seem to prove your >> point. The following are the durations of the pg_restore step of >> pg_upgrade: >> >> * 10k

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-11 Thread Matthias van de Meent
On Tue, 11 Jun 2024 at 04:01, Nathan Bossart wrote: > > On Fri, Jun 07, 2024 at 11:10:25AM +0200, Matthias van de Meent wrote: > > My primary concern isn't the IO, but the O(shared_buffers) that we > > have to go through during a checkpoint. As I mentioned upthread, it is > > reasonably possible

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-10 Thread Nathan Bossart
On Fri, Jun 07, 2024 at 11:10:25AM +0200, Matthias van de Meent wrote: > My primary concern isn't the IO, but the O(shared_buffers) that we > have to go through during a checkpoint. As I mentioned upthread, it is > reasonably possible the new cluster is already setup with a good > fraction of the

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-07 Thread Dilip Kumar
On Fri, Jun 7, 2024 at 2:40 PM Matthias van de Meent wrote: > > On Fri, 7 Jun 2024 at 10:28, Dilip Kumar wrote: > > > > On Fri, Jun 7, 2024 at 11:57 AM Matthias van de Meent > > wrote: > >> > >> On Fri, 7 Jun 2024 at 07:18, Dilip Kumar wrote: > >>> > >>> On Wed, Jun 5, 2024 at 10:59 PM

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-07 Thread Matthias van de Meent
On Fri, 7 Jun 2024 at 10:28, Dilip Kumar wrote: > > On Fri, Jun 7, 2024 at 11:57 AM Matthias van de Meent > wrote: >> >> On Fri, 7 Jun 2024 at 07:18, Dilip Kumar wrote: >>> >>> On Wed, Jun 5, 2024 at 10:59 PM Matthias van de Meent >>> wrote: >>> >>> I agree with you that we introduced the

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-07 Thread Dilip Kumar
On Fri, Jun 7, 2024 at 11:57 AM Matthias van de Meent wrote: > > On Fri, 7 Jun 2024 at 07:18, Dilip Kumar wrote: > > > > On Wed, Jun 5, 2024 at 10:59 PM Matthias van de Meent > > wrote: > >> > >> On Wed, 5 Jun 2024 at 18:47, Ranier Vilela wrote: > >>> > >>> Why not use it too, if not

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-07 Thread Matthias van de Meent
On Fri, 7 Jun 2024 at 07:18, Dilip Kumar wrote: > > On Wed, Jun 5, 2024 at 10:59 PM Matthias van de Meent > wrote: >> >> On Wed, 5 Jun 2024 at 18:47, Ranier Vilela wrote: >>> >>> Why not use it too, if not binary_upgrade? >> >> Because in the normal case (not during binary_upgrade) you don't

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-06 Thread Dilip Kumar
On Wed, Jun 5, 2024 at 10:59 PM Matthias van de Meent wrote: > > On Wed, 5 Jun 2024 at 18:47, Ranier Vilela wrote: > > > > Em ter., 4 de jun. de 2024 às 16:39, Nathan Bossart > > escreveu: > >> > >> I noticed that the "Restoring database schemas in the new cluster" part of > >> pg_upgrade can

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-05 Thread Nathan Bossart
On Wed, Jun 05, 2024 at 07:28:42PM +0200, Matthias van de Meent wrote: > As for "on my laptop", that sounds very reasonable, but could you > check the performance on systems with larger shared buffer > configurations? I'd imagine (but haven't checked) that binary upgrade > target systems may

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-05 Thread Matthias van de Meent
On Wed, 5 Jun 2024 at 18:47, Ranier Vilela wrote: > > Em ter., 4 de jun. de 2024 às 16:39, Nathan Bossart > escreveu: >> >> I noticed that the "Restoring database schemas in the new cluster" part of >> pg_upgrade can take a while if you have many databases, so I experimented >> with a couple

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-05 Thread Nathan Bossart
On Wed, Jun 05, 2024 at 01:47:09PM -0300, Ranier Vilela wrote: > Why not use it too, if not binary_upgrade? > > else > { > appendPQExpBuffer(creaQry, "CREATE DATABASE %s WITH TEMPLATE = template0 > STRATEGY = FILE_COPY", > qdatname); > } > > It seems to me that it also improves in any use.

Re: use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-05 Thread Ranier Vilela
Em ter., 4 de jun. de 2024 às 16:39, Nathan Bossart < nathandboss...@gmail.com> escreveu: > I noticed that the "Restoring database schemas in the new cluster" part of > pg_upgrade can take a while if you have many databases, so I experimented > with a couple different settings to see if there are

use CREATE DATABASE STRATEGY = FILE_COPY in pg_upgrade

2024-06-04 Thread Nathan Bossart
I noticed that the "Restoring database schemas in the new cluster" part of pg_upgrade can take a while if you have many databases, so I experimented with a couple different settings to see if there are any easy ways to speed it up. The FILE_COPY strategy for CREATE DATABASE helped quite