RE: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2023-01-11 Thread Yoshikazu Imai (Fujitsu)
> ; Heikki > Linnakangas ; pgsql-hackers@lists.postgresql.org; Thomas > Munro > Subject: Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints > > On Fri, Sep 2, 2022 at 5:25 PM Justin Pryzby wrote: > > > > On Tue, Aug 02, 2022 at 12:50:43PM -

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-09-02 Thread Dilip Kumar
On Fri, Sep 2, 2022 at 5:25 PM Justin Pryzby wrote: > > On Tue, Aug 02, 2022 at 12:50:43PM -0500, Justin Pryzby wrote: > > Also, if I understand correctly, this patch seems to assume that nobody is > > connected to the source database. But what's actually enforced is just that > > nobody *else*

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-09-02 Thread Justin Pryzby
On Tue, Aug 02, 2022 at 12:50:43PM -0500, Justin Pryzby wrote: > Also, if I understand correctly, this patch seems to assume that nobody is > connected to the source database. But what's actually enforced is just that > nobody *else* is connected. Is it any issue that the current DB can be used

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-18 Thread Robert Haas
On Wed, Aug 17, 2022 at 12:02 AM Dilip Kumar wrote: > > Although 0002 is formally a performance optimization, I'm inclined to > > think that if we're going to commit it, it should also be back-patched > > into v15, because letting the code diverge when we're not even out of > > beta yet seems

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-16 Thread Dilip Kumar
On Fri, Aug 12, 2022 at 6:33 PM Robert Haas wrote: > > On Thu, Aug 11, 2022 at 2:15 PM Robert Haas wrote: > > As far as I know, this 0001 addresses all outstanding comments and > > fixes the reported bug. > > > > Does anyone think otherwise? > > If they do, they're keeping quiet, so I committed

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-12 Thread Robert Haas
On Thu, Aug 11, 2022 at 2:15 PM Robert Haas wrote: > As far as I know, this 0001 addresses all outstanding comments and > fixes the reported bug. > > Does anyone think otherwise? If they do, they're keeping quiet, so I committed this and back-patched it to v15. Regarding 0002 -- should it,

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-11 Thread Robert Haas
On Wed, Aug 10, 2022 at 1:01 AM Dilip Kumar wrote: > Done, along with that, I have also got the hunk of smgropen and > smgrclose in ScanSourceDatabasePgClass() which I had in v1 patch[1]. > Because here we do not want to reuse the smgr of the pg_class again so > instead of closing at the end with

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-09 Thread Dilip Kumar
On Sun, Aug 7, 2022 at 9:47 AM Andres Freund wrote: > > Hi, > > On 2022-08-07 09:24:40 +0530, Dilip Kumar wrote: > > On Sat, Aug 6, 2022 at 9:36 PM Tom Lane wrote: > > > > > > Dilip Kumar writes: > > > > On Fri, Aug 5, 2022 at 10:43 AM Dilip Kumar > > > > wrote: > > > >> Yeah maybe it is not

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-06 Thread Andres Freund
Hi, On 2022-08-07 09:24:40 +0530, Dilip Kumar wrote: > On Sat, Aug 6, 2022 at 9:36 PM Tom Lane wrote: > > > > Dilip Kumar writes: > > > On Fri, Aug 5, 2022 at 10:43 AM Dilip Kumar wrote: > > >> Yeah maybe it is not necessary to close as these unowned smgr will > > >> automatically get closed

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-06 Thread Dilip Kumar
On Sat, Aug 6, 2022 at 9:36 PM Tom Lane wrote: > > Dilip Kumar writes: > > On Fri, Aug 5, 2022 at 10:43 AM Dilip Kumar wrote: > >> Yeah maybe it is not necessary to close as these unowned smgr will > >> automatically get closed on the transaction end. > > I do not think this is a great idea for

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-06 Thread Tom Lane
Dilip Kumar writes: > On Fri, Aug 5, 2022 at 10:43 AM Dilip Kumar wrote: >> Yeah maybe it is not necessary to close as these unowned smgr will >> automatically get closed on the transaction end. I do not think this is a great idea for the per-relation smgrs created during

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-06 Thread Tom Lane
Dilip Kumar writes: > PFA patches for different problems discussed in the thread > 0001 - Fix the problem of skipping the empty block and buffer lock on > source buffer > 0002 - Remove fake relcache entry (same as 0001-BugfixInWalLogCreateDB.patch) > 0003 - Optimization to avoid extending block

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-05 Thread Andres Freund
Hi, On 2022-08-04 19:14:08 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2022-08-04 18:05:25 -0400, Tom Lane wrote: > >> In any case, DROP DATABASE is far from the only place with a problem. > > > What other place has a database corrupting potential of this magnitude just > > because

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-05 Thread Dilip Kumar
On Fri, Aug 5, 2022 at 10:43 AM Dilip Kumar wrote: > > Yeah maybe it is not necessary to close as these unowned smgr will > automatically get closed on the transaction end. Actually the > previous person of the patch had both these comments fixed. The > reason for explicitly closing it is that

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-04 Thread Dilip Kumar
On Fri, Aug 5, 2022 at 2:59 AM Andres Freund wrote: > > Hi, > > On 2022-08-03 16:45:23 +0530, Dilip Kumar wrote: > > Another version of the patch which closes the smgr at the end using > > smgrcloserellocator() and I have also added a commit message. > > What's the motivation behind the explicit

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-04 Thread Dilip Kumar
On Fri, Aug 5, 2022 at 5:36 AM Andres Freund wrote: > > Hi, > > On August 4, 2022 4:20:16 PM PDT, Tom Lane wrote: > >Yeah, the assumption that P_NEW would automatically match the source block > >was making me itchy too. An explicit test-and-elog seems worth the > >cycles. > > Is there a good

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-04 Thread Dilip Kumar
On Fri, Aug 5, 2022 at 4:31 AM Tom Lane wrote: > > I wrote: > > While I'm at it, I would like to strenuously object to the current > > framing of CreateAndCopyRelationData as a general-purpose copying > > mechanism. > > And while I'm piling on, how is this bit in RelationCopyStorageUsingBuffer >

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-04 Thread Robert Haas
On Thu, Aug 4, 2022 at 7:11 PM Tom Lane wrote: > [pile^2] Also, what is the rationale for locking the target buffer > but not the source buffer? That seems pretty hard to justify from > here, even granting the assumption that we don't expect any other > processes to be interested in these

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-04 Thread Robert Haas
On Thu, Aug 4, 2022 at 7:01 PM Tom Lane wrote: > And while I'm piling on, how is this bit in RelationCopyStorageUsingBuffer > not completely broken? Ouch. That's pretty bad. -- Robert Haas EDB: http://www.enterprisedb.com

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-04 Thread Robert Haas
On Thu, Aug 4, 2022 at 6:02 PM Tom Lane wrote: > Robert Haas writes: > > I have reviewed this patch and I don't see a problem with it. However, > > it would be nice if Andres or someone else who understands this area > > well (Tom? Thomas?) would also review it, because I also reviewed > >

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-04 Thread Tom Lane
Andres Freund writes: > Is there a good reason to rely on P_NEW at all? Both from an efficiency > and robustness POV it seems like it'd be better to use smgrextend to > bulk extend just after smgrcreate() and then fill s_b u using RBM_ZERO > (or whatever it is called). That bulk smgrextend would

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-04 Thread Tom Lane
Andres Freund writes: > On August 4, 2022 4:11:13 PM PDT, Tom Lane wrote: >> [pile^2] Also, what is the rationale for locking the target buffer >> but not the source buffer? That seems pretty hard to justify from >> here, even granting the assumption that we don't expect any other >> processes

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-04 Thread Andres Freund
Hi, On August 4, 2022 4:20:16 PM PDT, Tom Lane wrote: >Yeah, the assumption that P_NEW would automatically match the source block >was making me itchy too. An explicit test-and-elog seems worth the >cycles. Is there a good reason to rely on P_NEW at all? Both from an efficiency and

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-04 Thread Andres Freund
Hi, On August 4, 2022 4:11:13 PM PDT, Tom Lane wrote: >I wrote: >> And while I'm piling on, how is this bit in RelationCopyStorageUsingBuffer >> not completely broken? > >[pile^2] Also, what is the rationale for locking the target buffer >but not the source buffer? That seems pretty hard to

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-04 Thread Tom Lane
Andres Freund writes: > On 2022-08-04 19:01:06 -0400, Tom Lane wrote: >> (This seems worth fixing before beta3, as it looks like a rather >> nasty data corruption hazard.) > Ugh, yes. And even with this fixed I think this should grow at least an > assertion that the block numbers match, probably

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-04 Thread Tom Lane
Andres Freund writes: > On 2022-08-04 18:05:25 -0400, Tom Lane wrote: >> In any case, DROP DATABASE is far from the only place with a problem. > What other place has a database corrupting potential of this magnitude just > because interrupts are accepted? We throw valid s_b contents away and

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-04 Thread Andres Freund
Hi, On 2022-08-04 19:01:06 -0400, Tom Lane wrote: > And while I'm piling on, how is this bit in RelationCopyStorageUsingBuffer > not completely broken? > > /* Read block from source relation. */ > srcBuf = ReadBufferWithoutRelcache(src->rd_locator, forkNum, blkno, >

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-04 Thread Tom Lane
I wrote: > And while I'm piling on, how is this bit in RelationCopyStorageUsingBuffer > not completely broken? [pile^2] Also, what is the rationale for locking the target buffer but not the source buffer? That seems pretty hard to justify from here, even granting the assumption that we don't

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-04 Thread Tom Lane
I wrote: > While I'm at it, I would like to strenuously object to the current > framing of CreateAndCopyRelationData as a general-purpose copying > mechanism. And while I'm piling on, how is this bit in RelationCopyStorageUsingBuffer not completely broken? /* Read block from source

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-04 Thread Andres Freund
Hi, On 2022-08-04 18:05:25 -0400, Tom Lane wrote: > Andres Freund writes: > > Yea. I think at the very least we need to start holding interrupts before > > the > > DropDatabaseBuffers() and do so until commit. That's probably best done by > > doing the transaction commit inside dropdb. > >

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-04 Thread Justin Pryzby
On Thu, Aug 04, 2022 at 06:02:50PM -0400, Tom Lane wrote: > The "invalidation" comment bothered me for awhile, but I think it's > fine: we know that no other backend can connect to the source DB > because we have it locked, About that - is it any problem that the currently-connected db can be

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-04 Thread Tom Lane
Andres Freund writes: > Yea. I think at the very least we need to start holding interrupts before the > DropDatabaseBuffers() and do so until commit. That's probably best done by > doing the transaction commit inside dropdb. We've talked before about ignoring interrupts across commit, but I find

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-04 Thread Tom Lane
Robert Haas writes: > I have reviewed this patch and I don't see a problem with it. However, > it would be nice if Andres or someone else who understands this area > well (Tom? Thomas?) would also review it, because I also reviewed > what's in the tree now and that turns out to be buggy, which

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-04 Thread Andres Freund
Hi, On 2022-08-04 16:38:35 +0530, Dilip Kumar wrote: > So basically, from this we can say it is completely a problem with > drop databases, I mean I can produce any behavior by interrupting drop > database > 1. If we created some tables/inserted data and the drop database got > cancelled, it

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-04 Thread Andres Freund
Hi, On 2022-08-04 16:07:01 -0400, Robert Haas wrote: > On Wed, Aug 3, 2022 at 7:15 AM Dilip Kumar wrote: > > Another version of the patch which closes the smgr at the end using > > smgrcloserellocator() and I have also added a commit message. > > I have reviewed this patch and I don't see a

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-04 Thread Andres Freund
Hi, On 2022-08-03 16:45:23 +0530, Dilip Kumar wrote: > Another version of the patch which closes the smgr at the end using > smgrcloserellocator() and I have also added a commit message. What's the motivation behind the explicit close? > @@ -258,8 +258,8 @@ ScanSourceDatabasePgClass(Oid tbid,

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-04 Thread Tom Lane
Justin Pryzby writes: > On Thu, Aug 04, 2022 at 04:07:01PM -0400, Robert Haas wrote: >> I'm inclined to hold off on committing this until next week, not only > +1 +1 ... there are some other v15 open items that I don't think we'll see fixed for beta3, either. regards,

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-04 Thread Justin Pryzby
On Thu, Aug 04, 2022 at 04:07:01PM -0400, Robert Haas wrote: > I'm inclined to hold off on committing this until next week, not only +1 I don't see any reason to hurry to fix problems that occur when DROP DATABASE is interrupted. Sorry to beat up your patches so much and for such crappy test

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-04 Thread Robert Haas
On Wed, Aug 3, 2022 at 7:15 AM Dilip Kumar wrote: > Another version of the patch which closes the smgr at the end using > smgrcloserellocator() and I have also added a commit message. I have reviewed this patch and I don't see a problem with it. However, it would be nice if Andres or someone

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-04 Thread Dilip Kumar
On Thu, Aug 4, 2022 at 9:41 AM Dilip Kumar wrote: > > On Thu, Aug 4, 2022 at 12:18 AM Justin Pryzby wrote: > > > > On Wed, Aug 03, 2022 at 11:26:43AM -0700, Andres Freund wrote: > > > Hm. This looks more like an issue of DROP DATABASE not being > > > interruptible. I > > > suspect this isn't

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-03 Thread Dilip Kumar
On Thu, Aug 4, 2022 at 12:18 AM Justin Pryzby wrote: > > On Wed, Aug 03, 2022 at 11:26:43AM -0700, Andres Freund wrote: > > Hm. This looks more like an issue of DROP DATABASE not being interruptible. > > I > > suspect this isn't actually related to STRATEGY wal_log and could likely be > >

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-03 Thread Justin Pryzby
On Wed, Aug 03, 2022 at 11:26:43AM -0700, Andres Freund wrote: > Hm. This looks more like an issue of DROP DATABASE not being interruptible. I > suspect this isn't actually related to STRATEGY wal_log and could likely be > reproduced in older versions too. I couldn't reproduce it with file_copy,

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-03 Thread Andres Freund
Hi, On 2022-08-03 12:01:18 -0500, Justin Pryzby wrote: > Now, I've reproduced the problem under valgrind, but it doesn't show anything > useful Yea, that looks like an issue on a different level. > > pryzbyj@pryzbyj:~$ while :; do psql -h /tmp template1 -c "DROP DATABASE a" -c > "CREATE

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-03 Thread Justin Pryzby
On Wed, Aug 03, 2022 at 11:02:00AM -0500, Justin Pryzby wrote: > But I reproduced the first problem with a handful of tries interrupting the > while loop: > > 2022-08-03 10:39:50.129 CDT client backend[5530] [unknown] PANIC: could not > open critical system index 2662 ... > So far, I haven't

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-03 Thread Dilip Kumar
On Wed, Aug 3, 2022 at 9:32 PM Justin Pryzby wrote: > > On Wed, Aug 03, 2022 at 04:45:23PM +0530, Dilip Kumar wrote: > > Another version of the patch which closes the smgr at the end using > > smgrcloserellocator() and I have also added a commit message. > > Thanks for providing a patch. > This

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-03 Thread Justin Pryzby
On Wed, Aug 03, 2022 at 04:45:23PM +0530, Dilip Kumar wrote: > Another version of the patch which closes the smgr at the end using > smgrcloserellocator() and I have also added a commit message. Thanks for providing a patch. This seems to fix the second problem with accessing freed memory. But I

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-03 Thread Dilip Kumar
On Wed, 3 Aug 2022 at 9:22 PM, Robert Haas wrote: > On Wed, Aug 3, 2022 at 7:15 AM Dilip Kumar wrote: > > Another version of the patch which closes the smgr at the end using > > smgrcloserellocator() and I have also added a commit message. > > Hmm, but didn't we decide against doing it that way

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-03 Thread Robert Haas
On Wed, Aug 3, 2022 at 7:15 AM Dilip Kumar wrote: > Another version of the patch which closes the smgr at the end using > smgrcloserellocator() and I have also added a commit message. Hmm, but didn't we decide against doing it that way intentionally? The comment you're deleting says "If we

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-03 Thread Dilip Kumar
On Wed, Aug 3, 2022 at 1:41 PM Dilip Kumar wrote: > > On Wed, Aug 3, 2022 at 12:00 PM Dilip Kumar wrote: > > > > > Okay, so AtEOXact_SMgr will only get rid of unowned smgr and ours are > > owned by a fake relcache and whose lifetime is just portal memory > > context which will go away at the

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-03 Thread Dilip Kumar
On Wed, Aug 3, 2022 at 12:00 PM Dilip Kumar wrote: > > Okay, so AtEOXact_SMgr will only get rid of unowned smgr and ours are > owned by a fake relcache and whose lifetime is just portal memory > context which will go away at the transaction end. So as Andres > suggested options could be that a)

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-03 Thread Dilip Kumar
On Wed, Aug 3, 2022 at 11:28 AM Dilip Kumar wrote: > > On Wed, Aug 3, 2022 at 3:53 AM Andres Freund wrote: > > > > On 2022-08-02 17:04:16 -0500, Justin Pryzby wrote: > > > I got this interesting looking thing. > > > > > > ==11628== Invalid write of size 8 > > > ==11628==at 0x1D12B3A:

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-02 Thread Dilip Kumar
On Wed, Aug 3, 2022 at 3:53 AM Andres Freund wrote: > > On 2022-08-02 17:04:16 -0500, Justin Pryzby wrote: > > I got this interesting looking thing. > > > > ==11628== Invalid write of size 8 > > ==11628==at 0x1D12B3A: smgrsetowner (smgr.c:213) > > ==11628==by 0x1C7C224: RelationGetSmgr

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-02 Thread Andres Freund
On 2022-08-02 17:04:16 -0500, Justin Pryzby wrote: > I got this interesting looking thing. > > ==11628== Invalid write of size 8 > ==11628==at 0x1D12B3A: smgrsetowner (smgr.c:213) > ==11628==by 0x1C7C224: RelationGetSmgr (rel.h:572) > ==11628==by 0x1C7C224:

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-02 Thread Justin Pryzby
On Tue, Aug 02, 2022 at 05:46:34PM -0400, Tom Lane wrote: > Robert Haas writes: > > WARNING: problem in alloc set PortalContext: req size > alloc size for > > chunk 0x7f99508911f0 in block 0x7f9950890800 > > This looks like nothing so much as the fallout from something scribbling > past the

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-02 Thread Tom Lane
Robert Haas writes: > WARNING: problem in alloc set PortalContext: req size > alloc size > for chunk 0x7f99508911f0 in block 0x7f9950890800 > WARNING: problem in alloc set PortalContext: bad size 0 for chunk > 0x7f99508911f0 in block 0x7f9950890800 > WARNING: problem in alloc set

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-02 Thread Robert Haas
On Tue, Aug 2, 2022 at 1:50 PM Justin Pryzby wrote: > Unfortunately, that isn't very consistent, and you have have to run it a bunch > of times... I was eventually able to reproduce this in part by using the interactive psql method you describe. It didn't crash, but it did spit out a bunch of

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-08-02 Thread Justin Pryzby
On Tue, Mar 29, 2022 at 11:55:05AM -0400, Robert Haas wrote: > On Mon, Mar 28, 2022 at 3:08 PM Robert Haas wrote: > > smgrcreate() as we would for most WAL records or whether it should be > > adopting the new system introduced by > > 49d9cfc68bf4e0d32a948fe72d5a0ef7f464944e. I wrote about this

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-04-04 Thread Dilip Kumar
On Sun, Apr 3, 2022 at 9:52 PM Andres Freund wrote: > > Hi, > > On 2022-03-29 11:55:05 -0400, Robert Haas wrote: > > I committed v6 instead. > > Coverity complains that this patch added GetDatabasePath() calls without > freeing its return value. Normally that'd be easy to dismiss, due to memory >

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-04-03 Thread Andres Freund
Hi, On 2022-03-29 11:55:05 -0400, Robert Haas wrote: > I committed v6 instead. Coverity complains that this patch added GetDatabasePath() calls without freeing its return value. Normally that'd be easy to dismiss, due to memory contexts, but there's no granular resets in

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-04-01 Thread Dilip Kumar
On Thu, Mar 31, 2022 at 9:52 PM Andres Freund wrote: > > + "ALTER DATABASE testdb SET TABLESPACE regress_ts2"); > > +ok($result == 0, 'move database to tablespace 2'); > > This just tests the command doesn't fail, but not whether it actually did > something useful. Seem we should at least

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-31 Thread Robert Haas
On Thu, Mar 31, 2022 at 2:44 PM Andres Freund wrote: > On 2022-03-31 14:31:43 -0400, Robert Haas wrote: > > On Thu, Mar 31, 2022 at 12:25 PM Andres Freund wrote: > > > > Andres, thoughts? Do you want me to polish and commit 0001? > > > > > > Yes please! > > > > Here is a polished version.

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-31 Thread Andres Freund
On 2022-03-31 14:31:43 -0400, Robert Haas wrote: > On Thu, Mar 31, 2022 at 12:25 PM Andres Freund wrote: > > > Andres, thoughts? Do you want me to polish and commit 0001? > > > > Yes please! > > Here is a polished version. Comments? LGTM.

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-31 Thread Robert Haas
On Thu, Mar 31, 2022 at 12:25 PM Andres Freund wrote: > > Andres, thoughts? Do you want me to polish and commit 0001? > > Yes please! Here is a polished version. Comments? > FWIW, once the freeze is done I'm planning to set up scripting to see which > parts of the code we whacked around don't

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-31 Thread Andres Freund
Hi, On 2022-03-31 10:05:10 -0400, Robert Haas wrote: > On Thu, Mar 31, 2022 at 3:52 AM Dilip Kumar wrote: > > 0001 is changing the strategy to file copy during initdb and 0002 > > patch adds the test cases for both these cases. > > IMHO, 0001 looks fine, except for needing some adjustments to

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-31 Thread Andres Freund
Hi, On 2022-03-31 13:22:24 +0530, Dilip Kumar wrote: > 0001 is changing the strategy to file copy during initdb and 0002 > patch adds the test cases for both these cases. Thanks! > From 4a997e2a95074a520777cd2b369f9c728b360969 Mon Sep 17 00:00:00 2001 > From: Dilip Kumar > Date: Thu, 31 Mar

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-31 Thread Robert Haas
On Thu, Mar 31, 2022 at 3:52 AM Dilip Kumar wrote: > 0001 is changing the strategy to file copy during initdb and 0002 > patch adds the test cases for both these cases. IMHO, 0001 looks fine, except for needing some adjustments to the wording. I'm less sure about 0002. It's testing the stuff

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-31 Thread Dilip Kumar
On Thu, Mar 31, 2022 at 9:46 AM Dilip Kumar wrote: > > On Thu, Mar 31, 2022 at 5:07 AM Andres Freund wrote: > > > > Hi, > > > > On 2022-03-29 11:55:05 -0400, Robert Haas wrote: > > > I committed v6 instead. > > > > I was just discussing the WAL prefetching patch with Thomas. A question in > >

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-30 Thread Dilip Kumar
On Thu, Mar 31, 2022 at 5:07 AM Andres Freund wrote: > > Hi, > > On 2022-03-29 11:55:05 -0400, Robert Haas wrote: > > I committed v6 instead. > > I was just discussing the WAL prefetching patch with Thomas. A question in > that discussion made me look at the coverage of REDO for CREATE DATABASE:

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-30 Thread Andres Freund
Hi, On 2022-03-29 11:55:05 -0400, Robert Haas wrote: > I committed v6 instead. I was just discussing the WAL prefetching patch with Thomas. A question in that discussion made me look at the coverage of REDO for CREATE DATABASE:

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-30 Thread Andres Freund
Hi, On 2022-03-30 09:28:58 +0530, Dilip Kumar wrote: > On Wed, Mar 30, 2022 at 6:47 AM Andres Freund wrote: > > > > > > du -s /tmp/initdb/ > > WAL_LOG: 35112 > > FILE_COPY: 29288 > > > > So it seems we should specify a strategy in initdb? It kind of makes sense - > > we're not going to read

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-30 Thread Dilip Kumar
On Tue, Mar 29, 2022 at 9:25 PM Robert Haas wrote: > > On Mon, Mar 28, 2022 at 3:08 PM Robert Haas wrote: > > smgrcreate() as we would for most WAL records or whether it should be > > adopting the new system introduced by > > 49d9cfc68bf4e0d32a948fe72d5a0ef7f464944e. I wrote about this concern >

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-29 Thread Dilip Kumar
On Wed, Mar 30, 2022 at 6:47 AM Andres Freund wrote: > > > du -s /tmp/initdb/ > WAL_LOG: 35112 > FILE_COPY: 29288 > > So it seems we should specify a strategy in initdb? It kind of makes sense - > we're not going to read anything from those database. And because of the > ringbuffer of 256kB,

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-29 Thread Andres Freund
Hi, On 2022-03-29 11:55:05 -0400, Robert Haas wrote: > I committed v6 instead. Just noticed that it makes initdb a bit slower / the cluster a bit bigger, because now there's WAL traffic from creating the databases. There's an optimization (albeit insufficient) to reduce WAL traffic in bootstrap

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-29 Thread Robert Haas
On Tue, Mar 29, 2022 at 2:37 PM Tom Lane wrote: > Robert Haas writes: > > On Tue, Mar 29, 2022 at 2:17 PM Tom Lane wrote: > >> it's some random algorithm that they probably feel at liberty > >> to change. > > > I guess that characterization surprises me. The man page for > > getopt_long() says

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-29 Thread Tom Lane
Robert Haas writes: > On Tue, Mar 29, 2022 at 2:17 PM Tom Lane wrote: >> it's some random algorithm that they probably feel at liberty >> to change. > I guess that characterization surprises me. The man page for > getopt_long() says this, and has for a long time at least on systems > I've used:

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-29 Thread Robert Haas
On Tue, Mar 29, 2022 at 2:17 PM Tom Lane wrote: > Robert Haas writes: > > On Tue, Mar 29, 2022 at 1:53 PM Tom Lane wrote: > >> That test script is expecting glibc-like laxness of switch > >> parsing. Put the switches before the non-switch arguments. > > > I just did that. :-) > > Yup, you

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-29 Thread Tom Lane
Robert Haas writes: > On Tue, Mar 29, 2022 at 1:53 PM Tom Lane wrote: >> That test script is expecting glibc-like laxness of switch >> parsing. Put the switches before the non-switch arguments. > I just did that. :-) Yup, you pushed while I was typing. FWIW, I don't think it's "Windows"

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-29 Thread Robert Haas
On Tue, Mar 29, 2022 at 1:53 PM Tom Lane wrote: > Andres Freund writes: > > Looks like there's some problem with commandline parsing? > > That test script is expecting glibc-like laxness of switch > parsing. Put the switches before the non-switch arguments. I just did that. :-) -- Robert

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-29 Thread Tom Lane
Andres Freund writes: > Looks like there's some problem with commandline parsing? That test script is expecting glibc-like laxness of switch parsing. Put the switches before the non-switch arguments. regards, tom lane

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-29 Thread Robert Haas
On Tue, Mar 29, 2022 at 1:35 PM Andres Freund wrote: > # Running: createdb -T foobar2 foobar6 -S wal_log > createdb: error: too many command-line arguments (first is "wal_log") > Try "createdb --help" for more information. > not ok 31 - createdb -T foobar2 foobar6 -S wal_log exit code 0 > > Looks

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-29 Thread Andres Freund
On 2022-03-29 11:55:05 -0400, Robert Haas wrote: > That commit having been reverted, I committed v6 instead. Let's see > what breaks... It fails in CI (for the mirror of the postgres repo on github): https://cirrus-ci.com/task/6279465603956736?logs=test_bin#L121 tap test log:

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-29 Thread Robert Haas
On Mon, Mar 28, 2022 at 3:08 PM Robert Haas wrote: > smgrcreate() as we would for most WAL records or whether it should be > adopting the new system introduced by > 49d9cfc68bf4e0d32a948fe72d5a0ef7f464944e. I wrote about this concern > over here: > >

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-28 Thread Dilip Kumar
On Tue, Mar 29, 2022 at 12:38 AM Robert Haas wrote: > > On Mon, Mar 28, 2022 at 2:18 AM Dilip Kumar wrote: > > > I have put the similar logic for relmap_update WAL replay as well, > > > > There was some mistake in the last patch, basically, for relmap update > > also I have checked the missing

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-28 Thread Robert Haas
On Mon, Mar 28, 2022 at 2:18 AM Dilip Kumar wrote: > > I have put the similar logic for relmap_update WAL replay as well, > > There was some mistake in the last patch, basically, for relmap update > also I have checked the missing tablespace directory but I should have > checked the missing

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-28 Thread Dilip Kumar
On Sat, Mar 26, 2022 at 5:55 PM Dilip Kumar wrote: > > On Fri, Mar 25, 2022 at 8:16 PM Dilip Kumar wrote: > > > > > On a quick look, I'm guessing that XLOG_DBASE_CREATE_WAL_LOG will need > > > to mirror some of the logic that was added to the replay code for the > > > existing strategy, but I

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-26 Thread Dilip Kumar
On Fri, Mar 25, 2022 at 8:16 PM Dilip Kumar wrote: > > > On a quick look, I'm guessing that XLOG_DBASE_CREATE_WAL_LOG will need > > to mirror some of the logic that was added to the replay code for the > > existing strategy, but I haven't figured out the details. > > > > Yeah, I think I got it,

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-25 Thread Dilip Kumar
On Fri, Mar 25, 2022 at 7:41 PM Robert Haas wrote: > > On Thu, Mar 24, 2022 at 12:12 PM Dilip Kumar wrote: > > Thanks, I have gone through your changes in comments and docs and those > > LGTM. > > It looks like this patch will need to be updated for Alvaro's commit >

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-25 Thread Robert Haas
On Thu, Mar 24, 2022 at 12:12 PM Dilip Kumar wrote: > Thanks, I have gone through your changes in comments and docs and those LGTM. It looks like this patch will need to be updated for Alvaro's commit 49d9cfc68bf4e0d32a948fe72d5a0ef7f464944e. The newly added test 029_replay_tsp_drops.pl fails

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-24 Thread Dilip Kumar
On Thu, Mar 24, 2022 at 9:29 PM Robert Haas wrote: > > On Thu, Mar 24, 2022 at 1:29 AM Dilip Kumar wrote: > > In the latest version I have fixed this issue by using a non > > conflicting name, because when it was compiled with-icu the foobar5 > > was already used and we were seeing failure.

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-24 Thread Robert Haas
On Thu, Mar 24, 2022 at 1:29 AM Dilip Kumar wrote: > In the latest version I have fixed this issue by using a non > conflicting name, because when it was compiled with-icu the foobar5 > was already used and we were seeing failure. Apart from this I have > fixed the duplicate cleanup problem by

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-23 Thread Dilip Kumar
On Wed, Mar 23, 2022 at 10:50 PM Dilip Kumar wrote: > > On Wed, Mar 23, 2022 at 10:37 PM Andres Freund wrote: > > > > Hi, > > > > On 2022-03-23 22:29:40 +0530, Dilip Kumar wrote: > > > I could not see any reason for it to fail, and I could not reproduce > > > it either. Is it possible to access

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-23 Thread Dilip Kumar
On Wed, Mar 23, 2022 at 10:37 PM Andres Freund wrote: > > Hi, > > On 2022-03-23 22:29:40 +0530, Dilip Kumar wrote: > > I could not see any reason for it to fail, and I could not reproduce > > it either. Is it possible to access the server log for this cfbot > > failure? > > Yes, near the top,

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-23 Thread Andres Freund
Hi, On 2022-03-23 22:29:40 +0530, Dilip Kumar wrote: > I could not see any reason for it to fail, and I could not reproduce > it either. Is it possible to access the server log for this cfbot > failure? Yes, near the top, below the cpu / memory graphs, there's a file navigator. Should have all

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-23 Thread Dilip Kumar
On Wed, Mar 23, 2022 at 9:25 PM Dilip Kumar wrote: > > On Wed, Mar 23, 2022 at 9:13 PM Andres Freund wrote: > > > > Hi, > > > > On 2022-03-23 18:49:11 +0530, Dilip Kumar wrote: > > > I think directly using smgrcreate() is a better idea instead of first > > > registering and then unregistering

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-23 Thread Dilip Kumar
On Wed, Mar 23, 2022 at 9:05 PM Dilip Kumar wrote: > > On Wed, Mar 23, 2022 at 7:03 PM Robert Haas wrote: > > > > On Wed, Mar 23, 2022 at 9:19 AM Dilip Kumar wrote: > > > I think directly using smgrcreate() is a better idea instead of first > > > registering and then unregistering it. I have

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-23 Thread Dilip Kumar
On Wed, Mar 23, 2022 at 9:13 PM Andres Freund wrote: > > Hi, > > On 2022-03-23 18:49:11 +0530, Dilip Kumar wrote: > > I think directly using smgrcreate() is a better idea instead of first > > registering and then unregistering it. I have made that change in > > the attached patch. After this

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-23 Thread Andres Freund
Hi, On 2022-03-23 18:49:11 +0530, Dilip Kumar wrote: > I think directly using smgrcreate() is a better idea instead of first > registering and then unregistering it. I have made that change in > the attached patch. After this change now we can merge creating the > MAIN_FORKNUM also in the loop

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-23 Thread Dilip Kumar
On Wed, Mar 23, 2022 at 7:03 PM Robert Haas wrote: > > On Wed, Mar 23, 2022 at 9:19 AM Dilip Kumar wrote: > > I think directly using smgrcreate() is a better idea instead of first > > registering and then unregistering it. I have made that change in > > the attached patch. After this change

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-23 Thread Robert Haas
On Wed, Mar 23, 2022 at 9:19 AM Dilip Kumar wrote: > I think directly using smgrcreate() is a better idea instead of first > registering and then unregistering it. I have made that change in > the attached patch. After this change now we can merge creating the > MAIN_FORKNUM also in the loop

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-23 Thread Dilip Kumar
On Wed, Mar 23, 2022 at 5:54 PM Robert Haas wrote: > > On Wed, Mar 23, 2022 at 4:42 AM Dilip Kumar wrote: > > Okay this is an interesting point. So one option is that in case of > > failure while using the wal log strategy we do not remove the database > > directory, because an abort

  1   2   3   >