Re: Avoiding smgrimmedsync() during nbtree index builds

2022-08-02 Thread Jacob Champion
This entry has been waiting on author input for a while (our current threshold is roughly two weeks), so I've marked it Returned with Feedback. Once you think the patchset is ready for review again, you (or any interested party) can resurrect the patch entry by visiting https://commitfest.pos

Re: Avoiding smgrimmedsync() during nbtree index builds

2022-07-23 Thread Heikki Linnakangas
On 05/03/2022 00:03, Melanie Plageman wrote: On Wed, Mar 2, 2022 at 8:09 PM Justin Pryzby wrote: Rebased to appease cfbot. I ran these paches under a branch which shows code coverage in cirrus. It looks good to my eyes. https://api.cirrus-ci.com/v1/artifact/task/5212346552418304/coverage/cov

Re: Avoiding smgrimmedsync() during nbtree index builds

2022-03-30 Thread Greg Stark
It looks like this patch received a review from Andres and hasn't been updated since. I'm not sure but the review looks to me like it's not ready to commit and needs some cleanup or at least to check on a few things. I guess it's not going to get bumped in the next few days so I'll move it to the

Re: Avoiding smgrimmedsync() during nbtree index builds

2022-03-10 Thread Andres Freund
Hi, > From a06407b19c8d168ea966e45c0e483b38d49ddc12 Mon Sep 17 00:00:00 2001 > From: Melanie Plageman > Date: Fri, 4 Mar 2022 14:48:39 -0500 > Subject: [PATCH v6 1/4] Add unbuffered IO API I think this or one of the following patches should update src/backend/access/transam/README > @@ -164,6

Re: Avoiding smgrimmedsync() during nbtree index builds

2022-03-04 Thread Melanie Plageman
On Wed, Mar 2, 2022 at 8:09 PM Justin Pryzby wrote: > > Rebased to appease cfbot. > > I ran these paches under a branch which shows code coverage in cirrus. It > looks good to my eyes. > https://api.cirrus-ci.com/v1/artifact/task/5212346552418304/coverage/coverage/00-index.html thanks for doing

Re: Avoiding smgrimmedsync() during nbtree index builds

2022-03-02 Thread Justin Pryzby
Rebased to appease cfbot. I ran these paches under a branch which shows code coverage in cirrus. It looks good to my eyes. https://api.cirrus-ci.com/v1/artifact/task/5212346552418304/coverage/coverage/00-index.html Are these patches being considered for v15 ? >From 30707da3e5eb68d1efbc5594696da4

Re: Avoiding smgrimmedsync() during nbtree index builds

2022-02-13 Thread Dmitry Dolgov
> On Wed, Feb 09, 2022 at 01:49:30PM -0500, Melanie Plageman wrote: > Hi, > v5 attached and all email feedback addressed below Thanks for the patch, it looks quite good. I don't see it in the discussion, so naturally curious -- why directmgr is not used for bloom index, e.g. in blbuildempty? > O

Re: Avoiding smgrimmedsync() during nbtree index builds

2022-02-09 Thread Melanie Plageman
Hi, v5 attached and all email feedback addressed below On Thu, Jan 13, 2022 at 12:18 PM Robert Haas wrote: > > On Wed, Sep 29, 2021 at 2:36 PM Melanie Plageman > wrote: > > unbuffered_write() and unbuffered_extend() might be able to be used even > > if unbuffered_prep() and unbuffered_finish() a

Re: Avoiding smgrimmedsync() during nbtree index builds

2022-01-23 Thread Andres Freund
Hi, On 2022-01-11 12:10:54 -0500, Melanie Plageman wrote: > On Mon, Jan 10, 2022 at 5:50 PM Melanie Plageman > wrote: > > > > I have attached a v3 which includes two commits -- one of which > > implements the directmgr API and uses it and the other which adds > > functionality to use either direc

Re: Avoiding smgrimmedsync() during nbtree index builds

2022-01-17 Thread Justin Pryzby
On Sun, Jan 16, 2022 at 02:25:59PM -0600, Justin Pryzby wrote: > On Thu, Jan 13, 2022 at 09:52:55AM -0600, Justin Pryzby wrote: > > This is failing on windows CI when I use initdb --data-checksums, as > > attached. > > > > https://cirrus-ci.com/task/5612464120266752 > > https://api.cirrus-ci.com/

Re: Avoiding smgrimmedsync() during nbtree index builds

2022-01-16 Thread Justin Pryzby
On Thu, Jan 13, 2022 at 09:52:55AM -0600, Justin Pryzby wrote: > This is failing on windows CI when I use initdb --data-checksums, as attached. > > https://cirrus-ci.com/task/5612464120266752 > https://api.cirrus-ci.com/v1/artifact/task/5612464120266752/regress_diffs/src/test/regress/regression.di

Re: Avoiding smgrimmedsync() during nbtree index builds

2022-01-13 Thread Robert Haas
On Wed, Sep 29, 2021 at 2:36 PM Melanie Plageman wrote: > unbuffered_write() and unbuffered_extend() might be able to be used even > if unbuffered_prep() and unbuffered_finish() are not used -- for example > hash indexes do something I don't entirely understand in which they call > smgrextend() di

Re: Avoiding smgrimmedsync() during nbtree index builds

2022-01-13 Thread Justin Pryzby
On Tue, Jan 11, 2022 at 12:10:54PM -0500, Melanie Plageman wrote: > On Mon, Jan 10, 2022 at 5:50 PM Melanie Plageman > wrote: > > > > I have attached a v3 which includes two commits -- one of which > > implements the directmgr API and uses it and the other which adds > > functionality to use eith

Re: Avoiding smgrimmedsync() during nbtree index builds

2022-01-11 Thread Melanie Plageman
On Mon, Jan 10, 2022 at 5:50 PM Melanie Plageman wrote: > > I have attached a v3 which includes two commits -- one of which > implements the directmgr API and uses it and the other which adds > functionality to use either directmgr or bufmgr API during index build. > > Also registering for march c

Re: Avoiding smgrimmedsync() during nbtree index builds

2022-01-10 Thread Melanie Plageman
I have attached a v3 which includes two commits -- one of which implements the directmgr API and uses it and the other which adds functionality to use either directmgr or bufmgr API during index build. Also registering for march commitfest. On Thu, Dec 9, 2021 at 2:33 PM Andres Freund wrote: > >

Re: Avoiding smgrimmedsync() during nbtree index builds

2021-12-09 Thread Andres Freund
Hi, On 2021-11-19 15:11:57 -0500, Melanie Plageman wrote: > From 2130175c5d794f60c5f15d6eb1b626c81fb7c39b Mon Sep 17 00:00:00 2001 > From: Melanie Plageman > Date: Thu, 15 Apr 2021 07:01:01 -0400 > Subject: [PATCH v2] Index build avoids immed fsync > > Avoid immediate fsync for just built indexe

Re: Avoiding smgrimmedsync() during nbtree index builds

2021-11-23 Thread Melanie Plageman
On Fri, Nov 19, 2021 at 3:11 PM Melanie Plageman wrote: > > On Mon, May 3, 2021 at 5:24 PM Melanie Plageman > wrote: > > > > So, I've written a patch which avoids doing the immediate fsync for > > index builds either by using shared buffers or by queueing sync requests > > for the checkpointer. I

Re: Avoiding smgrimmedsync() during nbtree index builds

2021-11-19 Thread Melanie Plageman
On Mon, May 3, 2021 at 5:24 PM Melanie Plageman wrote: > > So, I've written a patch which avoids doing the immediate fsync for > index builds either by using shared buffers or by queueing sync requests > for the checkpointer. If a checkpoint starts during the index build and > the backend is not u

Re: Avoiding smgrimmedsync() during nbtree index builds

2021-09-29 Thread Melanie Plageman
On Mon, May 3, 2021 at 5:24 PM Melanie Plageman wrote: > On Thu, Jan 21, 2021 at 5:51 PM Andres Freund wrote: > > On 2021-01-21 23:54:04 +0200, Heikki Linnakangas wrote: > > > On 21/01/2021 22:36, Andres Freund wrote: > > > > > > > > Thinking through the correctness of replacing smgrimmedsync() w

Re: Avoiding smgrimmedsync() during nbtree index builds

2021-05-03 Thread Melanie Plageman
So, I've written a patch which avoids doing the immediate fsync for index builds either by using shared buffers or by queueing sync requests for the checkpointer. If a checkpoint starts during the index build and the backend is not using shared buffers for the index build, it will need to do the fs

Re: Avoiding smgrimmedsync() during nbtree index builds

2021-01-21 Thread Andres Freund
Hi, On 2021-01-21 23:54:04 +0200, Heikki Linnakangas wrote: > On 21/01/2021 22:36, Andres Freund wrote: > > A quick hack (probably not quite correct!) to evaluate the benefit shows > > that the attached script takes 2m17.223s with the smgrimmedsync and > > 0m22.870s passing skipFsync=false to writ

Re: Avoiding smgrimmedsync() during nbtree index builds

2021-01-21 Thread Heikki Linnakangas
On 21/01/2021 22:36, Andres Freund wrote: Hi, Every nbtree index build currently does an smgrimmedsync at the end: /* * Read tuples in correct sort order from tuplesort, and load them into * btree leaves. */ static void _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2) .

Avoiding smgrimmedsync() during nbtree index builds

2021-01-21 Thread Andres Freund
Hi, Every nbtree index build currently does an smgrimmedsync at the end: /* * Read tuples in correct sort order from tuplesort, and load them into * btree leaves. */ static void _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2) ... /* * When we WAL-logged ind