RE: [PATCH] Speedup truncates of relation forks

2019-09-24 Thread Jamison, Kirk
On Tuesday, September 24, 2019 5:41 PM (GMT+9), Fujii Masao wrote: > On Thu, Sep 19, 2019 at 9:42 AM Jamison, Kirk > wrote: > > > > On Wednesday, September 18, 2019 8:38 PM, Fujii Masao wrote: > > > On Tue, Sep 17, 2019 at 10:44 AM Jamison, Kirk > > > > > > wrote: > > > > > > > > On Friday, Septe

Re: [PATCH] Speedup truncates of relation forks

2019-09-24 Thread Fujii Masao
On Thu, Sep 19, 2019 at 9:42 AM Jamison, Kirk wrote: > > On Wednesday, September 18, 2019 8:38 PM, Fujii Masao wrote: > > On Tue, Sep 17, 2019 at 10:44 AM Jamison, Kirk > > wrote: > > > > > > On Friday, September 13, 2019 10:06 PM (GMT+9), Fujii Masao wrote: > > > > On Fri, Sep 13, 2019 at 9:51 P

RE: [PATCH] Speedup truncates of relation forks

2019-09-18 Thread Jamison, Kirk
On Wednesday, September 18, 2019 8:38 PM, Fujii Masao wrote: > On Tue, Sep 17, 2019 at 10:44 AM Jamison, Kirk > wrote: > > > > On Friday, September 13, 2019 10:06 PM (GMT+9), Fujii Masao wrote: > > > On Fri, Sep 13, 2019 at 9:51 PM Alvaro Herrera > > > > > > wrote: > > > > > > > > On 2019-Sep-13,

Re: [PATCH] Speedup truncates of relation forks

2019-09-18 Thread Fujii Masao
On Tue, Sep 17, 2019 at 2:25 PM Michael Paquier wrote: > > On Tue, Sep 17, 2019 at 01:44:12AM +, Jamison, Kirk wrote: > > On Friday, September 13, 2019 10:06 PM (GMT+9), Fujii Masao wrote: > >> On Fri, Sep 13, 2019 at 9:51 PM Alvaro Herrera > >> wrote: > As committed, the smgrdounlinkfor

Re: [PATCH] Speedup truncates of relation forks

2019-09-18 Thread Fujii Masao
On Tue, Sep 17, 2019 at 10:44 AM Jamison, Kirk wrote: > > On Friday, September 13, 2019 10:06 PM (GMT+9), Fujii Masao wrote: > > On Fri, Sep 13, 2019 at 9:51 PM Alvaro Herrera > > wrote: > > > > > > On 2019-Sep-13, Fujii Masao wrote: > > > > > > > On Mon, Sep 9, 2019 at 3:52 PM Jamison, Kirk > >

Re: [PATCH] Speedup truncates of relation forks

2019-09-16 Thread Michael Paquier
On Tue, Sep 17, 2019 at 01:44:12AM +, Jamison, Kirk wrote: > On Friday, September 13, 2019 10:06 PM (GMT+9), Fujii Masao wrote: >> On Fri, Sep 13, 2019 at 9:51 PM Alvaro Herrera >> wrote: As committed, the smgrdounlinkfork case is actually dead code; it's never called from anywhere.

RE: [PATCH] Speedup truncates of relation forks

2019-09-16 Thread Jamison, Kirk
On Friday, September 13, 2019 10:06 PM (GMT+9), Fujii Masao wrote: > On Fri, Sep 13, 2019 at 9:51 PM Alvaro Herrera > wrote: > > > > On 2019-Sep-13, Fujii Masao wrote: > > > > > On Mon, Sep 9, 2019 at 3:52 PM Jamison, Kirk > wrote: > > > > > > > Please add a preliminary patch that removes the fun

Re: [PATCH] Speedup truncates of relation forks

2019-09-13 Thread Fujii Masao
On Fri, Sep 13, 2019 at 9:51 PM Alvaro Herrera wrote: > > On 2019-Sep-13, Fujii Masao wrote: > > > On Mon, Sep 9, 2019 at 3:52 PM Jamison, Kirk > > wrote: > > > > > Please add a preliminary patch that removes the function. Dead code is > > > > good, > > > > as long as it is gone. We can get i

Re: [PATCH] Speedup truncates of relation forks

2019-09-13 Thread Alvaro Herrera
On 2019-Sep-13, Fujii Masao wrote: > On Mon, Sep 9, 2019 at 3:52 PM Jamison, Kirk wrote: > > > Please add a preliminary patch that removes the function. Dead code is > > > good, > > > as long as it is gone. We can get it pushed ahead of the rest of this. > > > > Alright. I've attached a separ

Re: [PATCH] Speedup truncates of relation forks

2019-09-13 Thread Fujii Masao
On Mon, Sep 9, 2019 at 3:52 PM Jamison, Kirk wrote: > > On Friday, September 6, 2019 11:51 PM (GMT+9), Alvaro Herrera wrote: > > Hi Alvaro, > Thank you very much for the review! > > > On 2019-Sep-05, Jamison, Kirk wrote: > > > > > I also mentioned it from my first post if we can just remove this d

Re: [PATCH] Speedup truncates of relation forks

2019-09-13 Thread Fujii Masao
On Thu, Sep 5, 2019 at 5:53 PM Jamison, Kirk wrote: > > On Tuesday, September 3, 2019 9:44 PM (GMT+9), Fujii Masao wrote: > > Thanks for the patch! > > Thank you as well for the review! > > > -smgrdounlinkfork(SMgrRelation reln, ForkNumber forknum, bool isRedo) > > +smgrdounlinkfork(SMgrRelation r

RE: [PATCH] Speedup truncates of relation forks

2019-09-08 Thread Jamison, Kirk
On Friday, September 6, 2019 11:51 PM (GMT+9), Alvaro Herrera wrote: Hi Alvaro, Thank you very much for the review! > On 2019-Sep-05, Jamison, Kirk wrote: > > > I also mentioned it from my first post if we can just remove this dead code. > > If not, it would require to modify the function becaus

Re: [PATCH] Speedup truncates of relation forks

2019-09-06 Thread Alvaro Herrera from 2ndQuadrant
On 2019-Sep-05, Jamison, Kirk wrote: > I also mentioned it from my first post if we can just remove this dead code. > If not, it would require to modify the function because it would also > need nforks as input argument when calling DropRelFileNodeBuffers. I kept my > changes in the latest patch.

RE: [PATCH] Speedup truncates of relation forks

2019-09-05 Thread Jamison, Kirk
On Tuesday, September 3, 2019 9:44 PM (GMT+9), Fujii Masao wrote: > Thanks for the patch! Thank you as well for the review! > -smgrdounlinkfork(SMgrRelation reln, ForkNumber forknum, bool isRedo) > +smgrdounlinkfork(SMgrRelation reln, ForkNumber *forknum, int nforks, > bool isRedo) > > smgrdounl

Re: [PATCH] Speedup truncates of relation forks

2019-09-03 Thread Fujii Masao
On Wed, Jul 24, 2019 at 9:58 AM Jamison, Kirk wrote: > > Hi, > > I repeated the recovery performance test before, and found out that I made a > wrong measurement. > Using the same steps indicated in the previous email (24GB shared_buffers for > my case), > the recovery time still significantly im

RE: [PATCH] Speedup truncates of relation forks

2019-07-23 Thread Jamison, Kirk
Hi, I repeated the recovery performance test before, and found out that I made a wrong measurement. Using the same steps indicated in the previous email (24GB shared_buffers for my case), the recovery time still significantly improved compared to head from "13 minutes" to "4 minutes 44 seconds"

RE: [PATCH] Speedup truncates of relation forks

2019-07-08 Thread Jamison, Kirk
Hi Thomas, Thanks for checking. > On Fri, Jul 5, 2019 at 3:03 PM Jamison, Kirk wrote: > > I updated the patch which is similar to V3 of the patch, but > > addressing my problem in (5) in the previous email regarding > FreeSpaceMapVacuumRange. > > It seems to pass the regression test now. Kindly

Re: [PATCH] Speedup truncates of relation forks

2019-07-08 Thread Thomas Munro
On Fri, Jul 5, 2019 at 3:03 PM Jamison, Kirk wrote: > I updated the patch which is similar to V3 of the patch, > but addressing my problem in (5) in the previous email regarding > FreeSpaceMapVacuumRange. > It seems to pass the regression test now. Kindly check for validation. Hi Kirk, FYI ther

RE: [PATCH] Speedup truncates of relation forks

2019-07-04 Thread Jamison, Kirk
Hi, > I updated the patch based from comments, but it still fails the regression > test as indicated in (5) above. > Kindly verify if I correctly addressed the other parts as what you intended. > > Thanks again for the review! > I'll update the patch again after further comments. I updated the p

RE: [PATCH] Speedup truncates of relation forks

2019-07-04 Thread Jamison, Kirk
On Tuesday, July 2, 2019 4:59 PM (GMT+9), Masahiko Sawada wrote: > Thank you for updating the patch. Here is the review comments for v2 patch. Thank you so much for review! I indicated the addressed parts below and attached the updated patch. --- visibilitymap.c: visibilitymap_truncate() > I don

Re: [PATCH] Speedup truncates of relation forks

2019-07-03 Thread Adrien Nayrat
On 7/1/19 12:55 PM, Jamison, Kirk wrote: > On Wednesday, June 26, 2019 6:10 PM(GMT+9), Adrien Nayrat wrote: >> As far as I remember, you should see "relation" wait events (type lock) on >> standby server. This is due to startup process acquiring AccessExclusiveLock >> for the truncation and other b

Re: [PATCH] Speedup truncates of relation forks

2019-07-02 Thread Masahiko Sawada
On Mon, Jun 17, 2019 at 5:01 PM Jamison, Kirk wrote: > > Hi all, > > Attached is the v2 of the patch. I added the optimization that Sawada-san > suggested for DropRelFileNodeBuffers, although I did not acquire the lock > when comparing the minBlock and target block. > > There's actually a comment

RE: [PATCH] Speedup truncates of relation forks

2019-07-01 Thread Jamison, Kirk
On Wednesday, June 26, 2019 6:10 PM(GMT+9), Adrien Nayrat wrote: > As far as I remember, you should see "relation" wait events (type lock) on > standby server. This is due to startup process acquiring AccessExclusiveLock > for the truncation and other backend waiting to acquire a lock to read the >

Re: [PATCH] Speedup truncates of relation forks

2019-06-26 Thread Adrien Nayrat
On 6/12/19 10:29 AM, Jamison, Kirk wrote: > >> From a user POW, the main issue with relation truncation is that it can block >> queries on standby server during truncation replay. >> >> It could be interesting if you can test this case and give results of your >> path. >> Maybe by performing read

RE: [PATCH] Speedup truncates of relation forks

2019-06-17 Thread Jamison, Kirk
Hi all, Attached is the v2 of the patch. I added the optimization that Sawada-san suggested for DropRelFileNodeBuffers, although I did not acquire the lock when comparing the minBlock and target block. There's actually a comment written in the source code that we could pre-check buffer tag for f

RE: [PATCH] Speedup truncates of relation forks

2019-06-13 Thread Jamison, Kirk
Hi Sawada-san, On Thursday, June 13, 2019 8:01 PM, Masahiko Sawada wrote: > On Thu, Jun 13, 2019 at 6:30 PM Jamison, Kirk > wrote: > > > > On Wednesday, June 12, 2019 4:29 PM (GMT+9), Masahiko Sawada wrote: > > > ... > > > I've not look at this patch deeply but in DropRelFileNodeBuffer I > > > th

RE: [PATCH] Speedup truncates of relation forks

2019-06-13 Thread Tsunakawa, Takayuki
From: Masahiko Sawada [mailto:sawada.m...@gmail.com] > for (i = 0; i < NBuffers; i++) > { > (snip) > buf_state = LockBufHdr(bufHdr); > > /* check with the lower bound and skip the loop */ > if (bufHdr->tag.blockNum < minBlock) > { > UnlockBufHdr(

Re: [PATCH] Speedup truncates of relation forks

2019-06-13 Thread Masahiko Sawada
On Thu, Jun 13, 2019 at 6:30 PM Jamison, Kirk wrote: > > On Wednesday, June 12, 2019 4:29 PM (GMT+9), Masahiko Sawada wrote: > > On Wed, Jun 12, 2019 at 12:25 PM Tsunakawa, Takayuki > > wrote: > > > > > > From: Tomas Vondra [mailto:tomas.von...@2ndquadrant.com] > > > > Years ago I've implemented

RE: [PATCH] Speedup truncates of relation forks

2019-06-13 Thread Jamison, Kirk
On Wednesday, June 12, 2019 4:29 PM (GMT+9), Masahiko Sawada wrote: > On Wed, Jun 12, 2019 at 12:25 PM Tsunakawa, Takayuki > wrote: > > > > From: Tomas Vondra [mailto:tomas.von...@2ndquadrant.com] > > > Years ago I've implemented an optimization for many DROP TABLE > > > commands in a single trans

RE: [PATCH] Speedup truncates of relation forks

2019-06-12 Thread Tsunakawa, Takayuki
From: Masahiko Sawada [mailto:sawada.m...@gmail.com] > We do RelationTruncate() also when we truncate heaps that are created > in the current transactions or has a new relfilenodes in the current > transaction. So I think there is a room for optimization Thomas > suggested, although I'm not sure it

RE: [PATCH] Speedup truncates of relation forks

2019-06-12 Thread Jamison, Kirk
On Tuesday, June 11, 2019 7:23 PM (GMT+9), Adrien Nayrat wrote: > > Attached is a patch to speed up the performance of truncates of relations. > > Thanks for working on this! Thank you also for taking a look at my thread. > > If you want to test with large number of relations, > > you may use

Re: [PATCH] Speedup truncates of relation forks

2019-06-12 Thread Masahiko Sawada
On Wed, Jun 12, 2019 at 12:25 PM Tsunakawa, Takayuki wrote: > > From: Tomas Vondra [mailto:tomas.von...@2ndquadrant.com] > > Years ago I've implemented an optimization for many DROP TABLE commands > > in a single transaction - instead of scanning buffers for each relation, > > the code now accumul

RE: [PATCH] Speedup truncates of relation forks

2019-06-11 Thread Tsunakawa, Takayuki
From: Tomas Vondra [mailto:tomas.von...@2ndquadrant.com] > Years ago I've implemented an optimization for many DROP TABLE commands > in a single transaction - instead of scanning buffers for each relation, > the code now accumulates a small number of relations into an array, and > then does a bsear

Re: [PATCH] Speedup truncates of relation forks

2019-06-11 Thread Alvaro Herrera
On 2019-Jun-12, Tomas Vondra wrote: > Years ago I've implemented an optimization for many DROP TABLE commands > in a single transaction - instead of scanning buffers for each relation, > the code now accumulates a small number of relations into an array, and > then does a bsearch for each buffer.

Re: [PATCH] Speedup truncates of relation forks

2019-06-11 Thread Tomas Vondra
On Tue, Jun 11, 2019 at 07:34:35AM +, Jamison, Kirk wrote: Hi all, Attached is a patch to speed up the performance of truncates of relations. This is also my first time to contribute my own patch, and I'd gladly appreciate your feedback and advice. Thanks for the patch. Please add it to t

Re: [PATCH] Speedup truncates of relation forks

2019-06-11 Thread Adrien Nayrat
On 6/11/19 9:34 AM, Jamison, Kirk wrote: > Hi all, > > Attached is a patch to speed up the performance of truncates of relations. > Thanks for working on this! > > *C. **Performance Test* > > I setup a synchronous streaming replication between a master-standby. > > In postgresql.conf: >

[PATCH] Speedup truncates of relation forks

2019-06-11 Thread Jamison, Kirk
Hi all, Attached is a patch to speed up the performance of truncates of relations. This is also my first time to contribute my own patch, and I'd gladly appreciate your feedback and advice. A. Summary Whenever we truncate relations, it scans the shared buffers thrice (one per fork) which ca