Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-07-12 Thread Amit Kapila
On Thu, Jul 2, 2015 at 7:33 PM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: On 2 July 2015 at 14:08, Heikki Linnakangas hlinn...@iki.fi wrote: I'm marking this as returned with feedback in the commitfest. There are no unresolved issues with the approach,

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-07-02 Thread Heikki Linnakangas
On 07/02/2015 04:33 PM, Simon Riggs wrote: On 2 July 2015 at 14:08, Heikki Linnakangas hlinn...@iki.fi wrote: On 06/27/2015 07:45 AM, Amit Kapila wrote: Sometime back on one of the PostgreSQL blog [1], there was discussion about the performance of drop/truncate table for large values of

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-07-02 Thread Heikki Linnakangas
On 07/02/2015 04:18 PM, Amit Kapila wrote: On Thu, Jul 2, 2015 at 6:38 PM, Heikki Linnakangas hlinn...@iki.fi wrote: On 06/27/2015 07:45 AM, Amit Kapila wrote: Sometime back on one of the PostgreSQL blog [1], there was discussion about the performance of drop/truncate table for large values

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-07-02 Thread Andres Freund
On 2015-07-02 16:08:03 +0300, Heikki Linnakangas wrote: I'm marking this as returned with feedback in the commitfest. In addition to the issues raised so far, ISTM that the patch makes dropping a very large table with small shared_buffers slower (DropForkSpecificBuffers() is O(n) where n is

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-07-02 Thread Amit Kapila
On Thu, Jul 2, 2015 at 6:38 PM, Heikki Linnakangas hlinn...@iki.fi wrote: On 06/27/2015 07:45 AM, Amit Kapila wrote: Sometime back on one of the PostgreSQL blog [1], there was discussion about the performance of drop/truncate table for large values of shared_buffers and it seems that as the

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-07-02 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On 2 July 2015 at 14:08, Heikki Linnakangas hlinn...@iki.fi wrote: I'm marking this as returned with feedback in the commitfest. There are no unresolved issues with the approach, nor is it true it is slower. If you think there are some, you should say

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-07-02 Thread Simon Riggs
On 2 July 2015 at 14:08, Heikki Linnakangas hlinn...@iki.fi wrote: On 06/27/2015 07:45 AM, Amit Kapila wrote: Sometime back on one of the PostgreSQL blog [1], there was discussion about the performance of drop/truncate table for large values of shared_buffers and it seems that as the value

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-07-02 Thread Amit Kapila
On Thu, Jul 2, 2015 at 7:27 PM, Heikki Linnakangas hlinn...@iki.fi wrote: On 07/02/2015 04:18 PM, Amit Kapila wrote: Don't you think the approach discussed (delayed cleanup of buffers during checkpoint scan) is sufficient to fix the issues raised. Dunno, but there is no patch for that.

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-07-02 Thread Heikki Linnakangas
On 06/27/2015 07:45 AM, Amit Kapila wrote: Sometime back on one of the PostgreSQL blog [1], there was discussion about the performance of drop/truncate table for large values of shared_buffers and it seems that as the value of shared_buffers increase the performance of drop/truncate table

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-07-01 Thread Simon Riggs
On 1 July 2015 at 15:39, Amit Kapila amit.kapil...@gmail.com wrote: Okay. I think we can maintain the list in similar way as we do for UNLINK_RELATION_REQUEST in RememberFsyncRequest(), but why to wait till 64 tables? I meant once per checkpoint cycle OR every N tables, whichever is sooner.

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-07-01 Thread Amit Kapila
On Tue, Jun 30, 2015 at 12:10 PM, Simon Riggs si...@2ndquadrant.com wrote: On 30 June 2015 at 07:34, Amit Kapila amit.kapil...@gmail.com wrote: On Tue, Jun 30, 2015 at 11:00 AM, Simon Riggs si...@2ndquadrant.com wrote: On 30 June 2015 at 05:02, Amit Kapila amit.kapil...@gmail.com wrote:

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-07-01 Thread Amit Kapila
On Wed, Jul 1, 2015 at 8:26 PM, Simon Riggs si...@2ndquadrant.com wrote: On 1 July 2015 at 15:39, Amit Kapila amit.kapil...@gmail.com wrote: Okay. I think we can maintain the list in similar way as we do for UNLINK_RELATION_REQUEST in RememberFsyncRequest(), but why to wait till 64 tables?

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-06-30 Thread Amit Kapila
On Tue, Jun 30, 2015 at 11:00 AM, Simon Riggs si...@2ndquadrant.com wrote: On 30 June 2015 at 05:02, Amit Kapila amit.kapil...@gmail.com wrote: On Mon, Jun 29, 2015 at 7:18 PM, Simon Riggs si...@2ndquadrant.com wrote: On 28 June 2015 at 17:17, Tom Lane t...@sss.pgh.pa.us wrote: If

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-06-30 Thread Simon Riggs
On 30 June 2015 at 07:34, Amit Kapila amit.kapil...@gmail.com wrote: On Tue, Jun 30, 2015 at 11:00 AM, Simon Riggs si...@2ndquadrant.com wrote: On 30 June 2015 at 05:02, Amit Kapila amit.kapil...@gmail.com wrote: On Mon, Jun 29, 2015 at 7:18 PM, Simon Riggs si...@2ndquadrant.com wrote:

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-06-29 Thread Simon Riggs
On 28 June 2015 at 17:17, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: On 27 June 2015 at 15:10, Tom Lane t...@sss.pgh.pa.us wrote: I don't like this too much because it will fail badly if the caller is wrong about the maximum possible page number for the

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-06-29 Thread Amit Kapila
On Mon, Jun 29, 2015 at 7:18 PM, Simon Riggs si...@2ndquadrant.com wrote: On 28 June 2015 at 17:17, Tom Lane t...@sss.pgh.pa.us wrote: I'm not sure what you consider dire, but missing a dirty buffer belonging to the to-be-destroyed table would result in the system being permanently unable to

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-06-29 Thread Amit Kapila
On Mon, Jun 29, 2015 at 5:41 AM, Amit Kapila amit.kapil...@gmail.com wrote: On Sun, Jun 28, 2015 at 9:05 PM, Tom Lane t...@sss.pgh.pa.us wrote: Amit Kapila amit.kapil...@gmail.com writes: On Sat, Jun 27, 2015 at 7:40 PM, Tom Lane t...@sss.pgh.pa.us wrote: I don't like this too much

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-06-29 Thread Simon Riggs
On 30 June 2015 at 05:02, Amit Kapila amit.kapil...@gmail.com wrote: On Mon, Jun 29, 2015 at 7:18 PM, Simon Riggs si...@2ndquadrant.com wrote: On 28 June 2015 at 17:17, Tom Lane t...@sss.pgh.pa.us wrote: I'm not sure what you consider dire, but missing a dirty buffer belonging to the

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-06-28 Thread Robert Haas
On Sat, Jun 27, 2015 at 11:38 AM, Andres Freund and...@anarazel.de wrote: On 2015-06-27 10:10:04 -0400, Tom Lane wrote: In the past we've speculated about fixing the performance of these things by complicating the buffer lookup mechanism enough so that it could do find any page for this

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-06-28 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On 27 June 2015 at 15:10, Tom Lane t...@sss.pgh.pa.us wrote: I don't like this too much because it will fail badly if the caller is wrong about the maximum possible page number for the table, which seems not exactly far-fetched. (For instance,

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-06-28 Thread Andrew Dunstan
On 06/27/2015 10:10 AM, Tom Lane wrote: It also offers no hope of a fix for the other operations that scan the whole buffer pool, such as DROP TABLESPACE and DROP DATABASE. Improving DROP TABLE / TRUNCATE would still be a significant advance. These cases cause far more real world pain than

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-06-28 Thread Tom Lane
Amit Kapila amit.kapil...@gmail.com writes: On Sat, Jun 27, 2015 at 7:40 PM, Tom Lane t...@sss.pgh.pa.us wrote: I don't like this too much because it will fail badly if the caller is wrong about the maximum possible page number for the table, which seems not exactly far-fetched. (For

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-06-28 Thread Andres Freund
On 2015-06-28 09:11:29 -0400, Robert Haas wrote: On Sat, Jun 27, 2015 at 11:38 AM, Andres Freund and...@anarazel.de wrote: I've started to play around with doing that a year or three back. My approach was to use a linux style radix tree for the buffer mapping table. Besides lack of time

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-06-28 Thread Simon Riggs
On 27 June 2015 at 15:10, Tom Lane t...@sss.pgh.pa.us wrote: Amit Kapila amit.kapil...@gmail.com writes: I have looked into it and found that the main reason for such a behaviour is that for those operations it traverses whole shared_buffers and it seems to me that we don't need that

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-06-28 Thread Amit Kapila
On Sun, Jun 28, 2015 at 9:05 PM, Tom Lane t...@sss.pgh.pa.us wrote: Amit Kapila amit.kapil...@gmail.com writes: On Sat, Jun 27, 2015 at 7:40 PM, Tom Lane t...@sss.pgh.pa.us wrote: I don't like this too much because it will fail badly if the caller is wrong about the maximum possible page

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-06-28 Thread Amit Kapila
On Sun, Jun 28, 2015 at 9:47 PM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: On 27 June 2015 at 15:10, Tom Lane t...@sss.pgh.pa.us wrote: I don't like this too much because it will fail badly if the caller is wrong about the maximum possible page number for

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-06-28 Thread Robert Haas
On Sun, Jun 28, 2015 at 12:17 PM, Tom Lane t...@sss.pgh.pa.us wrote: I'm not sure what you consider dire, but missing a dirty buffer belonging to the to-be-destroyed table would result in the system being permanently unable to checkpoint, because attempts to write out the buffer to the

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-06-27 Thread Tom Lane
Amit Kapila amit.kapil...@gmail.com writes: I have looked into it and found that the main reason for such a behaviour is that for those operations it traverses whole shared_buffers and it seems to me that we don't need that especially for not-so-big tables. We can optimize that path by

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-06-27 Thread Gurjeet Singh
On Fri, Jun 26, 2015 at 9:45 PM, Amit Kapila amit.kapil...@gmail.com wrote: Sometime back on one of the PostgreSQL blog [1], there was discussion about the performance of drop/truncate table for large values of shared_buffers and it seems that as the value of shared_buffers increase the

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-06-27 Thread Amit Kapila
On Sat, Jun 27, 2015 at 8:06 PM, Gurjeet Singh gurj...@singh.im wrote: On Fri, Jun 26, 2015 at 9:45 PM, Amit Kapila amit.kapil...@gmail.com wrote: Attached patch implements the above idea and I found that performance doesn't dip much with patch even with large value of shared_buffers. I

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-06-27 Thread Amit Kapila
On Sat, Jun 27, 2015 at 7:40 PM, Tom Lane t...@sss.pgh.pa.us wrote: Amit Kapila amit.kapil...@gmail.com writes: I have looked into it and found that the main reason for such a behaviour is that for those operations it traverses whole shared_buffers and it seems to me that we don't need