Re: [PERFORM] Some performance testing?

2015-04-08 Thread Mel Llaguno
Cool. Good to know. I'll see if I can replicate these results in my 
environment. Thanks, M.


From: Josh Berkus 
Sent: Wednesday, April 08, 2015 1:05 PM
To: Mel Llaguno; Przemysław Deć
Cc: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Some performance testing?

On 04/07/2015 11:07 AM, Mel Llaguno wrote:
> Care to elaborate? We usually do not recommend specific kernel versions
> for our customers (who run on a variety of distributions). Thanks, M.

You should.

http://www.databasesoup.com/2014/09/why-you-need-to-avoid-linux-kernel-32.html

Performance is literally 2X to 5X different between kernels.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Some performance testing?

2015-04-07 Thread Mel Llaguno
Care to elaborate? We usually do not recommend specific kernel versions
for our customers (who run on a variety of distributions). Thanks, M.

Mel Llaguno • Staff Engineer – Team Lead
Office: +1.403.264.9717 x310
www.coverity.com <http://www.coverity.com/> • Twitter: @coverity
Coverity by Synopsys


On 4/7/15, 11:41 AM, "Josh Berkus"  wrote:

>On 04/07/2015 09:46 AM, Mel Llaguno wrote:
>> FYI - all my tests were conducted using Ubuntu 12.04 x64 LTS (which I
>> believe are all 3.xx series kernels).
>
>If it's 3.2 or 3.5, then your tests aren't useful, I'm afraid.  Both of
>those kernels have known, severe, memory management issues.
>
>-- 
>Josh Berkus
>PostgreSQL Experts Inc.
>http://pgexperts.com


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Some performance testing?

2015-04-07 Thread Mel Llaguno
FYI - all my tests were conducted using Ubuntu 12.04 x64 LTS (which I
believe are all 3.xx series kernels).

Mel Llaguno • Staff Engineer – Team Lead
Office: +1.403.264.9717 x310
www.coverity.com <http://www.coverity.com/> • Twitter: @coverity
Coverity by Synopsys




On 4/6/15, 2:51 PM, "Josh Berkus"  wrote:

>On 04/01/2015 01:37 AM, Przemysław Deć wrote:
>> Maybe you will find time to benchamark xfs vs ext4 (with and without
>> journaling enabled on ext4).
>> 
>> Nice comparison also could be rhel 6.5 with its newest kernel 2.6.32-X
>> vs RHEL 7.0 and kernel 3.10.
>
>Due to how these are hosted, I can't swap out kernels.
>
>-- 
>Josh Berkus
>PostgreSQL Experts Inc.
>http://pgexperts.com


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Some performance testing?

2015-04-03 Thread Mel Llaguno
Although the results only focus on SATA2 HDD, these may be useful for a 
comparison of ext4 vs. xfs : 
http://www.fuzzy.cz/bench/compare-pgbench.php?type[]=btrfs-datacow-barrier:1&type[]=btrfs-datacow-nobarrier:1&type[]=btrfs-nodatacow-barrier:1&type[]=btrfs-nodatacow-nobarrier:1&type[]=ext4-writeback-barrier:1&type[]=ext4-writeback-nobarrier:1&type[]=xfs-barrier:1&type[]=xfs-nobarrier:1?


M.


From: Przemysław Deć 
Sent: Wednesday, April 01, 2015 3:02 AM
To: Mel Llaguno
Cc: Josh Berkus; pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Some performance testing?

Maybe you will find time to benchamark xfs vs ext4 (with and without journaling 
enabled on ext4).

Nice comparison also could be rhel 6.5 with its newest kernel 2.6.32-X vs RHEL 
7.0 and kernel 3.10.

I was looking for some guidance what to choose and there is very poor 
information about such things.
[https://ssl.gstatic.com/ui/v1/icons/mail/images/cleardot.gif]
--
Przemysław Deć
Senior Solutions Architect
Linux Polska Sp. z o.o

2015-04-01 10:37 GMT+02:00 Przemysław Deć 
mailto:przemyslaw@linuxpolska.pl>>:
Maybe you will find time to benchamark xfs vs ext4 (with and without journaling 
enabled on ext4).

Nice comparison also could be rhel 6.5 with its newest kernel 2.6.32-X vs RHEL 
7.0 and kernel 3.10.

I was looking for some guidance what to choose and there is very poor 
information about such things.

--
Przemysław Deć
Senior Solutions Architect
Linux Polska Sp. z o.o


2015-03-31 22:41 GMT+02:00 Mel Llaguno 
mailto:mllag...@coverity.com>>:
It would be interesting to get raw performance benchmarks in addition to
PG specific benchmarks. I've been measuring raw I/O performance of a few
of our systems and run the following tests as well:

1. 10 runs of bonnie++
2. 5 runs of hdparm -Tt
3. Using a temp file created on the SSD, dd if=tempfile of=/dev/null bs=1M
count=1024 && echo 3 > /proc/sys/vm/drop_caches; dd
if=tempfileof=/dev/null bs=1M count=1024
4. Using phoronix benchmarks -> stream / ramspeed / compress-7zip


I was curious to measure the magnitude of difference between HDD -> SSD. I
would expect significant differences between SSD -> PCI-E Flash.

I've included some metrics from some previous runs vs. different types of
SSDs (OWC Mercury Extreme 6G which is our standard SSD, an Intel S3700
SSD, a Samsung SSD 840 PRO) vs. some standard HDD from  Western Digital
and HGST. I put in a req for a 960Gb Mercury Excelsior PCI-E SSD which
hasn't yet materialized ...

Thanks, M.

Mel Llaguno * Staff Engineer - Team Lead
Office: +1.403.264.9717 x310
www.coverity.com<http://www.coverity.com> <http://www.coverity.com/> * Twitter: 
@coverity
Coverity by Synopsys


On 3/31/15, 1:52 PM, "Josh Berkus" 
mailto:j...@agliodbs.com>> wrote:

>All,
>
>I currently have access to a matched pair of 20-core, 128GB RAM servers
>with SSD-PCI storage, for about 2 weeks before they go into production.
> Are there any performance tests people would like to see me run on
>these?  Otherwise, I'll just do some pgbench and DVDStore.
>
>--
>Josh Berkus
>PostgreSQL Experts Inc.
>http://pgexperts.com
>
>
>--
>Sent via pgsql-performance mailing list 
>(pgsql-performance@postgresql.org<mailto:pgsql-performance@postgresql.org>)
>To make changes to your subscription:
>http://www.postgresql.org/mailpref/pgsql-performance



--
Sent via pgsql-performance mailing list 
(pgsql-performance@postgresql.org<mailto:pgsql-performance@postgresql.org>)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance





Re: [PERFORM] Some performance testing?

2015-03-31 Thread Mel Llaguno
It would be interesting to get raw performance benchmarks in addition to
PG specific benchmarks. I’ve been measuring raw I/O performance of a few
of our systems and run the following tests as well:

1. 10 runs of bonnie++
2. 5 runs of hdparm -Tt
3. Using a temp file created on the SSD, dd if=tempfile of=/dev/null bs=1M
count=1024 && echo 3 > /proc/sys/vm/drop_caches; dd
if=tempfileof=/dev/null bs=1M count=1024
4. Using phoronix benchmarks -> stream / ramspeed / compress-7zip


I was curious to measure the magnitude of difference between HDD -> SSD. I
would expect significant differences between SSD -> PCI-E Flash.

I’ve included some metrics from some previous runs vs. different types of
SSDs (OWC Mercury Extreme 6G which is our standard SSD, an Intel S3700
SSD, a Samsung SSD 840 PRO) vs. some standard HDD from  Western Digital
and HGST. I put in a req for a 960Gb Mercury Excelsior PCI-E SSD which
hasn’t yet materialized ...

Thanks, M.

Mel Llaguno • Staff Engineer – Team Lead
Office: +1.403.264.9717 x310
www.coverity.com <http://www.coverity.com/> • Twitter: @coverity
Coverity by Synopsys


On 3/31/15, 1:52 PM, "Josh Berkus"  wrote:

>All,
>
>I currently have access to a matched pair of 20-core, 128GB RAM servers
>with SSD-PCI storage, for about 2 weeks before they go into production.
> Are there any performance tests people would like to see me run on
>these?  Otherwise, I'll just do some pgbench and DVDStore.
>
>-- 
>Josh Berkus
>PostgreSQL Experts Inc.
>http://pgexperts.com
>
>
>-- 
>Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
>To make changes to your subscription:
>http://www.postgresql.org/mailpref/pgsql-performance



System Benchmarks.xlsx
Description: System Benchmarks.xlsx

-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


[PERFORM] Anyone have experience using PG on a NetApp All-Flash FAS8000?

2015-03-13 Thread Mel Llaguno
The reason I ask is that it seems to support deduplication/compression. I was 
wondering if this would have any performance implications of PG operations.


Thanks, M.?


Mel Llaguno * Staff Engineer - Team Lead
Office: +1.403.264.9717 x310
www.coverity.com<http://www.coverity.com/> <http://www.coverity.com/> * 
Twitter: @coverity
Coverity by Synopsys


Re: [PERFORM] HFS+ pg_test_fsync performance

2014-04-23 Thread Mel Llaguno
Josh,

Thanks for the feedback. Given the prevalence of SSDs/VMs, it would be
useful to start collecting stats/tuning for different operating systems
for things like fsync (and possibly bonnie++/dd). If the community is
interested, I¹ve got a performance lab that I¹d be willing to help run
tests on. Having this information would only improve our ability to
support our customers.

M.

On 4/23/14, 12:04 PM, "Josh Berkus"  wrote:

>Mel,
>
>> I was given anecdotal information regarding HFS+ performance under OSX
>>as
>> being unsuitable for production PG deployments and that pg_test_fsync
>> could be used to measure the relative speed versus other operating
>>systems
>
>You're welcome to identify your source of anecdotal evidence: me.  And
>it's based on experience and testing, although I'm not allowed to share
>the raw results.  Let's just say that I was part of two different
>projects where we moved from using OSX on Apple hardware do using Linux
>on the *same* hardware ... because of intolerable performance on OSX.
>Switching to Linux more than doubled our real write throughput.
>
>> Compare file sync methods using one 8kB write:
>> (in wal_sync_method preference order, except fdatasync
>> is Linux's default)
>> open_datasync8752.240 ops/sec 114
>>usecs/op
>> fdatasync8556.469 ops/sec 117
>>usecs/op
>> fsync8831.080 ops/sec 113
>>usecs/op
>==
>==
>> fsync_writethrough735.362 ops/sec1360
>>usecs/op
>==
>==
>> open_sync8967.000 ops/sec 112
>>usecs/op
>
>fsync_writethrough is the *only* relevant stat above.  For all of the
>other fsync operations, OSX is "faking it"; returning to the calling
>code without ever flushing to disk.  This would result in data
>corruption in the event of an unexpected system shutdown.
>
>Both OSX and Windows do this, which is why we *have* fsync_writethrough.
> Mind you, I'm a little shocked that performance is still so bad on
>SSDs; I'd attributed HFS's slow fsync mostly to waiting for a full disk
>rotation, but apparently the primary cause is something else.
>
>You'll notice that the speed of fsync_writethrough is 1/4 that of
>comparable speed on Linux.   You can get similar performance on Linux by
>putting your WAL on a ramdisk, but I don't recommend that for production.
>
>But: things get worse.  In addition to the very slow speed on real
>fsyncs, HFS+ has very primitive IO scheduling for multiprocessor
>workloads; the filesystem was designed for single-core machines (in
>1995!) and has no ability to interleave writes from multiple concurrent
>processes.  This results in "stuttering" as the IO system tries to
>service first one write request, then another, and ends up stalling
>both.  If you do, for example, a serious ETL workload with parallelism
>on OSX, you'll see that IO throughput describes a sawtooth from full
>speed to zero, being near-zero about half the time.
>
>So not only are fsyncs slower, real throughput for sustained writes on
>HFS+ are 50% or less of the hardware maximum in any real multi-user
>workload.
>
>In order to test this, you'd need a workload which required loading and
>sorting several tables larger than RAM, at least two in parallel.
>
>In the words of the lead HFS+ developer, Don Brady:  "Since we believed
>it was only a stop gap solution, we just went from 16 to 32 bits. Had we
>known that it would still be in use 15 years later with multi-terabyte
>drives, we probably would have done more design changes!"
>
>HFS+ was written in about 6 months, and is largely unimproved since its
>release in 1995.  Ext2 doesn't perform too well, either; the difference
>is that Linux users have alternative filesystems available.
>
>-- 
>Josh Berkus
>PostgreSQL Experts Inc.
>http://pgexperts.com
>
>
>-- 
>Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
>To make changes to your subscription:
>http://www.postgresql.org/mailpref/pgsql-performance



-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] HFS+ pg_test_fsync performance

2014-04-15 Thread Mel Llaguno


My 2 cents :

The results are not surprising, in the linux enviroment the i/o call of 
pg_test_fsync  are using O_DIRECT  (PG_O_DIRECT) with also the O_SYNC or 
O_DSYNC calls, so ,in practice, it is waiting the "answer" from the storage 
bypassing the cache  in sync mode, while in  the Mac OS X it is not doing so, 
it's only using the O_SYNC or O_DSYNC calls without O_DIRECT,  in practice, 
it's using the cache of filesystem , even if it is asking the sync of io calls.


Bye

Mat Dba



Thanks for the explanation. Given that OSX always seems to use filesystem 
cache, is there a way to measure fsync performance that is equivalent to Linux? 
Or will the use of pg_test_fsync always be inflated under OSX? The reason I ask 
is that we would like to make a case with a customer that PG performance on 
OSX/HFS+ would be sub-optimal compared to using Linux/EXT4 (or FreeBSD/UFS2 for 
that matter).

Thanks, Mel


[PERFORM] HFS+ pg_test_fsync performance

2014-04-14 Thread Mel Llaguno
I was given anecdotal information regarding HFS+ performance under OSX as
being unsuitable for production PG deployments and that pg_test_fsync
could be used to measure the relative speed versus other operating systems
(such as Linux). In my performance lab, I have a number of similarly
equipped Linux hosts (Ubuntu 12.04 64-bit LTS Server w/128Gb RAM / 2 OWC
6g Mercury Extreme SSDs / 7200rpm SATA3 HDD / 16 E5-series cores) that I
used to capture baseline Linux numbers. As we generally recommend our
customers use SSD (the s3700 recommended by PG), I wanted to perform a
comparison. On these beefy machines I ran the following tests:

SSD:

# pg_test_fsync -f ./fsync.out -s 30
30 seconds per test
O_DIRECT supported on this platform for open_datasync and open_sync.

Compare file sync methods using one 8kB write:
(in wal_sync_method preference order, except fdatasync
is Linux's default)
open_datasync n/a
fdatasync2259.652 ops/sec 443 usecs/op
fsync1949.664 ops/sec 513 usecs/op
fsync_writethroughn/a
open_sync2245.162 ops/sec 445 usecs/op

Compare file sync methods using two 8kB writes:
(in wal_sync_method preference order, except fdatasync
is Linux's default)
open_datasync n/a
fdatasync2161.941 ops/sec 463 usecs/op
fsync1891.894 ops/sec 529 usecs/op
fsync_writethroughn/a
open_sync1118.826 ops/sec 894 usecs/op

Compare open_sync with different write sizes:
(This is designed to compare the cost of writing 16kB
in different write open_sync sizes.)
 1 * 16kB open_sync write2171.558 ops/sec 460 usecs/op
 2 *  8kB open_sync writes   1126.490 ops/sec 888 usecs/op
 4 *  4kB open_sync writes569.594 ops/sec1756 usecs/op
 8 *  2kB open_sync writes285.149 ops/sec3507 usecs/op
16 *  1kB open_sync writes142.528 ops/sec7016 usecs/op

Test if fsync on non-write file descriptor is honored:
(If the times are similar, fsync() can sync data written
on a different descriptor.)
write, fsync, close  1947.557 ops/sec 513 usecs/op
write, close, fsync  1951.082 ops/sec 513 usecs/op

Non-Sync'ed 8kB writes:
write   481296.909 ops/sec   2 usecs/op


HDD:

pg_test_fsync -f /tmp/fsync.out -s 30
30 seconds per test
O_DIRECT supported on this platform for open_datasync and open_sync.

Compare file sync methods using one 8kB write:
(in wal_sync_method preference order, except fdatasync
is Linux's default)
open_datasync n/a
fdatasync 105.783 ops/sec9453 usecs/op
fsync  27.692 ops/sec   36111 usecs/op
fsync_writethroughn/a
open_sync 103.399 ops/sec9671 usecs/op

Compare file sync methods using two 8kB writes:
(in wal_sync_method preference order, except fdatasync
is Linux's default)
open_datasync n/a
fdatasync 104.647 ops/sec9556 usecs/op
fsync  27.223 ops/sec   36734 usecs/op
fsync_writethroughn/a
open_sync  55.839 ops/sec   17909 usecs/op

Compare open_sync with different write sizes:
(This is designed to compare the cost of writing 16kB
in different write open_sync sizes.)
 1 * 16kB open_sync write 103.581 ops/sec9654 usecs/op
 2 *  8kB open_sync writes 55.207 ops/sec   18113 usecs/op
 4 *  4kB open_sync writes 28.320 ops/sec   35311 usecs/op
 8 *  2kB open_sync writes 14.581 ops/sec   68582 usecs/op
16 *  1kB open_sync writes  7.407 ops/sec  135003 usecs/op

Test if fsync on non-write file descriptor is honored:
(If the times are similar, fsync() can sync data written
on a different descriptor.)
write, fsync, close27.228 ops/sec   36727 usecs/op
write, close, fsync27.108 ops/sec   36890 usecs/op

Non-Sync'ed 8kB writes:
write   466108.001 ops/sec   2 usecs/op


---

So far, so good. Local HDD vs. SSD shows a significant difference in fsync
performance. Here are the corresponding fstab entries :

/dev/mapper/cim-base
/opt/cimext4defaults,noatime,nodiratime,discard 0   
2 (SSD)
/dev/mapper/p--app--lin-root /   ext4errors=remount-ro 0
1 (HDD)

I then tried the pg_test_fsync on my OSX Mavericks machine (quad-core i7 /
Intel 520SS

Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-09-09 Thread Mel Llaguno
Bruce,

First of all, I'd like to thank you for taking some interest in this issue. 
We'd love to migrate to the latest PG version, but this issue is currently 
preventing us from doing so.

Regardless of the DB used (base application schema _or_ restored DB with 
additional app data + base application schema), seed information is present in 
all tests. I guess my question is this - why would having existing data change 
the bind behavior at all? Is it possible that the way indexes are created has 
changed between 8.4 -> 9.x? 

Thanks, M.

Mel Llaguno | Principal Engineer (Performance/Deployment)
Coverity | 800 6th Avenue S.W. | Suite 410 | Calgary, AB | Canada | T2P 3G3
mllag...@coverity.com

From: Bruce Momjian
[br...@momjian.us]
Sent: Monday, September 09, 2013 8:16 PM
To: Mel
Llaguno
Cc: pgsql-performance@postgresql.org
Subject: Re: [PERFORM]
Performance bug in prepared statement binding in 9.2?

On Tue, Sep 10, 2013
at 02:06:08AM +, Mel Llaguno wrote:
> We're currently using an embedded
PG 8.4.17 for our application. Our
> PG 9.x tests consists of one of the
following :
>
> - Take a 8.4.17 DB which contains only our application
schema and
> required seed data and use pg_upgrade to create a 9.x
database
> directory, run the analyze_new_cluster.sh script and fire up
the
> application. Under this set of conditions, the bind connection issue
>
is present during our test.
>
> - Start with a fresh PG 9.x DB (using use
create_db) and use psql
> to recreate our application schema and required
seed data. When the
> application is started and our test executed, the bind
connection
> issue is still present.
>
> In both of the above cases, a base
application schema is used.
>
> If I upgrade an 8.4.17 DB that contains
additional application data
> (generated by interaction with our
application) to 9.x, the bind
> connection issue is no longer present.
Restoring this upgraded 9.x DB
> into any PG instance in the previously
described scenarios also seems
> to fix the bind connection issue.
>
>
Please let me know if this clarifies my earlier post.

Yes, that is clear.
So it is the seed data that is causing the problem?
That is the only
different I see from #2 and #3.

--
  Bruce Momjian  
http://momjian.us
  EnterpriseDB
http://enterprisedb.com

  + It's impossible for everything to be true. +






-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-09-09 Thread Mel Llaguno
We're currently using an embedded PG 8.4.17 for our application. Our PG 9.x 
tests consists of one of the following :

- Take a 8.4.17 DB which contains only our application schema and required seed 
data and use pg_upgrade to create a 9.x database directory, run the 
analyze_new_cluster.sh script and fire up the application. Under this set of 
conditions, the bind connection issue is present during our test.

- Start with a fresh PG 9.x DB (using use create_db) and use psql to recreate 
our application schema and required seed data. When the application is started 
and our test executed, the bind connection issue is still present.

In both of the above cases, a base application schema is used. 

If I upgrade an 8.4.17 DB that contains additional application data (generated 
by interaction with our application) to 9.x, the bind connection issue is no 
longer present. Restoring this upgraded 9.x DB into any PG instance in the 
previously described scenarios also seems to fix the bind connection issue.

Please let me know if this clarifies my earlier post. 

Thanks, M.

Mel Llaguno | Principal Engineer (Performance/Deployment)
Coverity | 800 6th Avenue S.W. | Suite 410 | Calgary, AB | Canada | T2P 3G3
mllag...@coverity.com

From: Bruce Momjian [br...@momjian.us]
Sent: Monday, September 09, 2013 7:45 PM
To: Mel Llaguno
Cc: Andrew Dunstan; Jeff Janes; Josh Berkus; Amit Kapila;
pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Performance bug in prepared statement binding in
9.2?

On Tue, Sep 10, 2013 at 01:36:27AM +0000, Mel Llaguno wrote:
> Let me clarify further - when we reconstruct our schema (without the
> upgrade step) via a sql script, the problem still persists. Restoring
> an upgraded DB which contains existing data into exactly the same
> instance will correct the behavior.

I do not understand what you are saying above.

--
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +






-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Performance bug in prepared statement binding in 9.2?

2013-09-09 Thread Mel Llaguno
Let me clarify further - when we reconstruct our schema (without the upgrade 
step) via a sql script, the problem still persists. Restoring an upgraded DB 
which contains existing data into exactly the same instance will correct the 
behavior. 



Mel Llaguno | Principal Engineer (Performance/Deployment)
Coverity | 800 6th Avenue S.W. | Suite 410 | Calgary, AB | Canada | T2P 3G3
mllag...@coverity.com


From: pgsql-performance-ow...@postgresql.org
[pgsql-performance-ow...@postgresql.org] on behalf of Andrew Dunstan
[and...@dunslane.net]
Sent: Monday, September 09, 2013 6:38 PM
To: Jeff Janes
Cc: Josh Berkus; Amit Kapila; pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Performance bug in prepared statement binding in
9.2?

On 08/01/2013 03:20 PM, Jeff Janes wrote:
> On Thu, Aug 1, 2013 at 10:58 AM, Josh Berkus  wrote:
>> Amit, All:
>>
>> So we just retested this on 9.3b2.  The performance is the same as 9.1
>> and 9.2; that is, progressively worse as the test cycles go on, and
>> unacceptably slow compared to 8.4.
>>
>> Some issue introduced in 9.1 is causing BINDs to get progressively
>> slower as the PARSEs BINDs get run repeatedly.  Per earlier on this
>> thread, that can bloat to 200X time required for a BIND, and it's
>> definitely PostgreSQL-side.
>>
>> I'm trying to produce a test case which doesn't involve the user's
>> application.  However, hints on other things to analyze would be keen.
> Does it seem to be all CPU time (it is hard to imagine what else it
> would be, but...)
>
> Could you use oprofile or perf or gprof to get a profile of the
> backend during a run?  That should quickly narrow it down to which C
> function has the problem.
>
> Did you test 9.0 as well?


This has been tested back to 9.0. What we have found is that the problem
disappears if the database has come in via dump/restore, but is present
if it is the result of pg_upgrade. There are some long-running
transactions also running alongside this - we are currently planning a
test where those are not present. We're also looking at constructing a
self-contained test case.

Here is some perf output from the bad case:

+  14.67%  postgres   [.] heap_hot_search_buffer
+  11.45%  postgres   [.] LWLockAcquire
+   8.39%  postgres   [.] LWLockRelease
+   6.60%  postgres   [.] _bt_checkkeys
+   6.39%  postgres   [.] PinBuffer
+   5.96%  postgres   [.] hash_search_with_hash_value
+   5.43%  postgres   [.] hash_any
+   5.14%  postgres   [.] UnpinBuffer
+   3.43%  postgres   [.] ReadBuffer_common
+   2.34%  postgres   [.] index_fetch_heap
+   2.04%  postgres   [.] heap_page_prune_opt
+   2.00%  libc-2.15.so   [.] 0x8041b
+   1.94%  postgres   [.] _bt_next
+   1.83%  postgres   [.] btgettuple
+   1.76%  postgres   [.] index_getnext_tid
+   1.70%  postgres   [.] LockBuffer
+   1.54%  postgres   [.] ReadBufferExtended
+   1.25%  postgres   [.] FunctionCall2Coll
+   1.14%  postgres   [.] HeapTupleSatisfiesNow
+   1.09%  postgres   [.] ReleaseAndReadBuffer
+   0.94%  postgres   [.] ResourceOwnerForgetBuffer
+   0.81%  postgres   [.] _bt_saveitem
+   0.80%  postgres   [.] _bt_readpage
+   0.79%  [kernel.kallsyms]  [k] 0x81170861
+   0.64%  postgres   [.] CheckForSerializableConflictOut
+   0.60%  postgres   [.] ResourceOwnerEnlargeBuffers
+   0.59%  postgres   [.] BufTableLookup

and here is the good case:

+   9.54%  libc-2.15.so   [.] 0x15eb1f
+   7.31%  [kernel.kallsyms]  [k] 0x8117924b
+   5.65%  postgres   [.] AllocSetAlloc
+   3.57%  postgres   [.] SearchCatCache
+   2.67%  postgres   [.] hash_search_with_hash_value
+   1.69%  postgres   [.] base_yyparse
+   1.49%  libc-2.15.so   [.] vfprintf
+   1.34%  postgres   [.] MemoryContextAllocZeroAligned
+   1.34%  postgres   [.] XLogInsert
+   1.24%  postgres   [.] copyObject
+   1.10%  postgres   [.] palloc
+   1.09%  postgres   [.] _bt_compare
+   1.04%  postgres   [.] core_yylex
+   0.96%  postgres   [.] _bt_checkkeys
+   0.95%  postgres   [.] expression_tree_walker
+   0.88%  postgres   [.] ScanKeywordLookup
+   0.87%  postgres   [.] pg_encoding_mbcliplen
+   0.86%  postgres   [.] LWLockAcquire
+   0.72%  postgres   [.] nocachegetattr
+   0.67%  postgres   [.] FunctionCall2Coll
+   0.63%  postgres   [.] fmgr_info_cxt_security
+   0.62%  postgres