Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2014-11-10 Thread Josh Berkus
On 12/31/2013 09:55 AM, Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: Tom, There's an abbreviated version of this argument in the comments in my proposed patch at http://www.postgresql.org/message-id/11927.1384199...@sss.pgh.pa.us What I'm hoping will happen next is that the

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2014-11-10 Thread Jeff Janes
On Mon, Nov 10, 2014 at 10:48 AM, Josh Berkus j...@agliodbs.com wrote: On 12/31/2013 09:55 AM, Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: Tom, There's an abbreviated version of this argument in the comments in my proposed patch at

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2014-11-10 Thread Josh Berkus
On 11/10/2014 10:59 AM, Jeff Janes wrote: On Mon, Nov 10, 2014 at 10:48 AM, Josh Berkus j...@agliodbs.com wrote: On 12/31/2013 09:55 AM, Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: Tom, There's an abbreviated version of this argument in the comments in my proposed patch at

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2014-11-10 Thread Andres Freund
On 2014-11-10 10:48:24 -0800, Josh Berkus wrote: On 12/31/2013 09:55 AM, Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: Tom, There's an abbreviated version of this argument in the comments in my proposed patch at http://www.postgresql.org/message-id/11927.1384199...@sss.pgh.pa.us

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2014-11-10 Thread Tom Lane
Josh Berkus j...@agliodbs.com writes: On 11/10/2014 10:59 AM, Jeff Janes wrote: On Mon, Nov 10, 2014 at 10:48 AM, Josh Berkus j...@agliodbs.com wrote: Did this patch every make it in? Or did it hang waiting for verification? It made it in: commit 4162a55c77cbb54acb4ac442ef3565b813b9d07a

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2014-11-10 Thread Josh Berkus
On 11/10/2014 11:11 AM, Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: On 11/10/2014 10:59 AM, Jeff Janes wrote: On Mon, Nov 10, 2014 at 10:48 AM, Josh Berkus j...@agliodbs.com wrote: Did this patch every make it in? Or did it hang waiting for verification? It made it in: commit

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2014-11-10 Thread Jeff Janes
On Mon, Nov 10, 2014 at 11:04 AM, Josh Berkus j...@agliodbs.com wrote: On 11/10/2014 10:59 AM, Jeff Janes wrote: On Mon, Nov 10, 2014 at 10:48 AM, Josh Berkus j...@agliodbs.com wrote: Did this patch every make it in? Or did it hang waiting for verification? It made it in:

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2014-11-10 Thread Josh Berkus
On 11/10/2014 12:13 PM, Jeff Janes wrote: The related problem where the end rows are actually needed (e.g. ORDER BY...LIMIT) has not been fixed. My idea to fix that was to check if the row's creation-transaction was in the MVCC snapshot (which just uses local memory) before checking if that

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-12-31 Thread Tom Lane
Josh Berkus j...@agliodbs.com writes: Tom, There's an abbreviated version of this argument in the comments in my proposed patch at http://www.postgresql.org/message-id/11927.1384199...@sss.pgh.pa.us What I'm hoping will happen next is that the complainants will hot-patch that and see if it

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-11-13 Thread Josh Berkus
Tom, There's an abbreviated version of this argument in the comments in my proposed patch at http://www.postgresql.org/message-id/11927.1384199...@sss.pgh.pa.us What I'm hoping will happen next is that the complainants will hot-patch that and see if it fixes their problems. We can't really

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-11-13 Thread Tom Lane
Josh Berkus j...@agliodbs.com writes: What I'm hoping will happen next is that the complainants will hot-patch that and see if it fixes their problems. We can't really determine what to do without that information. Unfortunately, the original reporter of this issue will not be available for

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-11-12 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: I wonder if we could ameliorate this problem by making get_actual_variable_range() use SnapshotDirty In that thread I claimed that a current MVCC snapshot was the most appropriate thing, which it probably is; If it reads from the end of the index, won't it

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-11-12 Thread Tom Lane
Kevin Grittner kgri...@ymail.com writes: Tom Lane t...@sss.pgh.pa.us wrote: I wonder if we could ameliorate this problem by making get_actual_variable_range() use SnapshotDirty In that thread I claimed that a current MVCC snapshot was the most appropriate thing, which it probably is; If it

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-11-08 Thread Tom Lane
Josh Berkus j...@agliodbs.com writes: The explanation is in http://archives.postgresql.org/message-id/20130910132133.GJ1024477%40alap2.anarazel.de The referenced commit introduced a planner feature. Funnily you seem to have been the trigger for it's introduction ;) Oh, crap, the off the

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-09-25 Thread Jeff Janes
On Wed, Sep 11, 2013 at 2:10 PM, Andres Freund and...@2ndquadrant.comwrote: On 2013-09-11 15:06:23 -0400, Andrew Dunstan wrote: One thing that this made me wonder is why we don't have transaction_timeout, or maybe transaction_idle_timeout. Because it's harder than it sounds, at least if

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-09-11 Thread Josh Berkus
All, We've confirmed that this issue is caused by having long-running idle transactions on the server. When we disabled their queueing system (which prodiced hour-long idle txns), the progressive slowness went away. Why that should affect 9.X far more strongly than 8.4, I'm not sure about.

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-09-11 Thread Andrew Dunstan
On 09/11/2013 02:35 PM, Josh Berkus wrote: All, We've confirmed that this issue is caused by having long-running idle transactions on the server. When we disabled their queueing system (which prodiced hour-long idle txns), the progressive slowness went away. Why that should affect 9.X far

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-09-11 Thread Andres Freund
On 2013-09-11 15:06:23 -0400, Andrew Dunstan wrote: On 09/11/2013 02:35 PM, Josh Berkus wrote: All, We've confirmed that this issue is caused by having long-running idle transactions on the server. When we disabled their queueing system (which prodiced hour-long idle txns), the

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-09-11 Thread Andres Freund
On 2013-09-11 11:35:45 -0700, Josh Berkus wrote: All, We've confirmed that this issue is caused by having long-running idle transactions on the server. When we disabled their queueing system (which prodiced hour-long idle txns), the progressive slowness went away. Why that should affect

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-09-11 Thread Josh Berkus
The explanation is in http://archives.postgresql.org/message-id/20130910132133.GJ1024477%40alap2.anarazel.de The referenced commit introduced a planner feature. Funnily you seem to have been the trigger for it's introduction ;) Oh, crap, the off the end of the index optimization? It's the

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-09-10 Thread Bruce Momjian
On Tue, Sep 10, 2013 at 05:43:16AM +, Mel Llaguno wrote: Bruce, First of all, I'd like to thank you for taking some interest in this issue. We'd love to migrate to the latest PG version, but this issue is currently preventing us from doing so. Regardless of the DB used (base application

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-09-10 Thread Andres Freund
On 2013-09-09 20:38:09 -0400, Andrew Dunstan wrote: On 08/01/2013 03:20 PM, Jeff Janes wrote: On Thu, Aug 1, 2013 at 10:58 AM, Josh Berkus j...@agliodbs.com wrote: Amit, All: So we just retested this on 9.3b2. The performance is the same as 9.1 and 9.2; that is, progressively worse as

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-09-10 Thread Andres Freund
Hi, On 2013-09-09 20:38:09 -0400, Andrew Dunstan wrote: On 08/01/2013 03:20 PM, Jeff Janes wrote: On Thu, Aug 1, 2013 at 10:58 AM, Josh Berkus j...@agliodbs.com wrote: Amit, All: So we just retested this on 9.3b2. The performance is the same as 9.1 and 9.2; that is, progressively worse

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-09-10 Thread Andrew Dunstan
On 09/10/2013 08:20 AM, Andres Freund wrote: A backtrace for this would be useful. Alternatively you could recompile postgres using -fno-omit-frame-pointer in CFLAGS and use perf record -g. It's using a custom build, so this should be doable. Any chance you have older prepared xacts,

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-09-10 Thread Andres Freund
On 2013-09-10 08:45:33 -0400, Andrew Dunstan wrote: On 09/10/2013 08:20 AM, Andres Freund wrote: A backtrace for this would be useful. Alternatively you could recompile postgres using -fno-omit-frame-pointer in CFLAGS and use perf record -g. It's using a custom build, so this should be

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-09-10 Thread Andres Freund
On 2013-09-10 15:21:33 +0200, Andres Freund wrote: If I interpret things correctly you're using serializable? I guess there is no chance to use repeatable read instead? Err, that wouldn't help much. Read committed. That lets PGXACT-xmin advance these days and thus might help to reduce the

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-09-10 Thread Andrew Dunstan
On 09/10/2013 09:23 AM, Andres Freund wrote: On 2013-09-10 15:21:33 +0200, Andres Freund wrote: If I interpret things correctly you're using serializable? I guess there is no chance to use repeatable read instead? Err, that wouldn't help much. Read committed. That lets PGXACT-xmin advance

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-09-09 Thread Andrew Dunstan
On 08/01/2013 03:20 PM, Jeff Janes wrote: On Thu, Aug 1, 2013 at 10:58 AM, Josh Berkus j...@agliodbs.com wrote: Amit, All: So we just retested this on 9.3b2. The performance is the same as 9.1 and 9.2; that is, progressively worse as the test cycles go on, and unacceptably slow compared to

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-09-09 Thread Bruce Momjian
On Mon, Sep 9, 2013 at 08:38:09PM -0400, Andrew Dunstan wrote: On 08/01/2013 03:20 PM, Jeff Janes wrote: On Thu, Aug 1, 2013 at 10:58 AM, Josh Berkus j...@agliodbs.com wrote: Amit, All: So we just retested this on 9.3b2. The performance is the same as 9.1 and 9.2; that is, progressively

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-09-09 Thread Mel Llaguno
: Monday, September 09, 2013 6:38 PM To: Jeff Janes Cc: Josh Berkus; Amit Kapila; pgsql-performance@postgresql.org Subject: Re: [PERFORM] Performance bug in prepared statement binding in 9.2? On 08/01/2013 03:20 PM, Jeff Janes wrote: On Thu, Aug 1, 2013 at 10:58 AM, Josh Berkus j...@agliodbs.com wrote

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-09-09 Thread Bruce Momjian
On Tue, Sep 10, 2013 at 01:36:27AM +, Mel Llaguno wrote: Let me clarify further - when we reconstruct our schema (without the upgrade step) via a sql script, the problem still persists. Restoring an upgraded DB which contains existing data into exactly the same instance will correct the

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-09-09 Thread Mel Llaguno
] Performance bug in prepared statement binding in 9.2? On Tue, Sep 10, 2013 at 01:36:27AM +, Mel Llaguno wrote: Let me clarify further - when we reconstruct our schema (without the upgrade step) via a sql script, the problem still persists. Restoring an upgraded DB which contains existing data

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-09-09 Thread Bruce Momjian
On Tue, Sep 10, 2013 at 02:06:08AM +, Mel Llaguno wrote: We're currently using an embedded PG 8.4.17 for our application. Our PG 9.x tests consists of one of the following : - Take a 8.4.17 DB which contains only our application schema and required seed data and use pg_upgrade to create a

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-09-09 Thread Mel Llaguno
] Performance bug in prepared statement binding in 9.2? On Tue, Sep 10, 2013 at 02:06:08AM +, Mel Llaguno wrote: We're currently using an embedded PG 8.4.17 for our application. Our PG 9.x tests consists of one of the following : - Take a 8.4.17 DB which contains only our application schema

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-08-01 Thread Josh Berkus
Amit, All: So we just retested this on 9.3b2. The performance is the same as 9.1 and 9.2; that is, progressively worse as the test cycles go on, and unacceptably slow compared to 8.4. Some issue introduced in 9.1 is causing BINDs to get progressively slower as the PARSEs BINDs get run

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-08-01 Thread Jeff Janes
On Thu, Aug 1, 2013 at 10:58 AM, Josh Berkus j...@agliodbs.com wrote: Amit, All: So we just retested this on 9.3b2. The performance is the same as 9.1 and 9.2; that is, progressively worse as the test cycles go on, and unacceptably slow compared to 8.4. Some issue introduced in 9.1 is

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-05-31 Thread Amit Kapila
On Thursday, May 30, 2013 11:36 PM Josh Berkus wrote: Amit, I think it might be because of choosing custom plan option due to which it might be generating new plan during exec_bind_message(). exec_bind_message()-GetCachedPlan()-choose_custom_plan(). If it chooses custom plan, then it will

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-05-31 Thread Josh Berkus
When you say interactively, does it mean that you are using psql to test the same? Yes. Of course, on the psql command line, there's no separate BIND step, just PREPARE and EXECUTE. From the changes, it doesn't seem that any of such changes can cause the problem you have seen. No, but

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-05-30 Thread Amit Kapila
On Thursday, May 30, 2013 5:45 AM Josh Berkus wrote: Folks, I'm seeing what may be a major performance bug in BIND in 9.2.4. We have a client who has an application which uses Tomcat+Hibernate+JDBC. They are in the process of upgrading this application from 8.4.17 to 9.2.4. As part of

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-05-30 Thread Josh Berkus
Amit, I think it might be because of choosing custom plan option due to which it might be generating new plan during exec_bind_message(). exec_bind_message()-GetCachedPlan()-choose_custom_plan(). If it chooses custom plan, then it will regenerate the plan which can cause extra cost

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-05-30 Thread Josh Berkus
This is why I'm asking them to run tests on 9.1. If 9.1 doesn't exhibit this behavior, then customplan is liable to be at fault. 9.1 shows the same performance as 9.2. So it's not the custom plan thing. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via

[PERFORM] Performance bug in prepared statement binding in 9.2?

2013-05-29 Thread Josh Berkus
Folks, I'm seeing what may be a major performance bug in BIND in 9.2.4. We have a client who has an application which uses Tomcat+Hibernate+JDBC. They are in the process of upgrading this application from 8.4.17 to 9.2.4. As part of this, they have been doing performance testing, and 9.2 is

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-05-29 Thread Josh Berkus
On 05/29/2013 05:14 PM, Josh Berkus wrote: Here's timings overall for 29 test cycles (cycle 1 has been omitted). As you can see, parse+execute times are pretty much constant, as are application think times, but bind times vary quite a lot. In 8.4, the 29 cycles are constantly 4.5min to