Re: [HACKERS] Managing multiple branches in git
On Sun, 7 Jun 2009, Peter Eisentraut wrote: On Wednesday 03 June 2009 01:55:48 Andrew Dunstan wrote: Running recursive grep on a subversion working copy is quite nasty. I suggest export GREP_OPTIONS='-d skip -I --exclude=*.svn-base --exclude=tags --exclude=*~ --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.svn --exclude=TAGS' The other alternative is to use ack: http://betterthangrep.com/ and have some better defaults. -- * Greg Smith gsm...@gregsmith.com http://www.gregsmith.com Baltimore, MD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 open item: copy performance regression?
It doesn't look like it's related to autovacuum. I re-ran the test against the two solaris boxes with autovacuum turned off and the results look about the same. 8.3.7 - Solaris 10 11/06 s10x_u3wos_10 X86 real0m43.662s user0m0.001s sys 0m0.003s real0m43.565s user0m0.001s sys 0m0.003s real0m43.742s user0m0.001s sys 0m0.003s 8.4rc1 - Solaris 10 11/06 s10x_u3wos_10 X86 real0m59.304s user0m0.001s sys 0m0.003s real0m58.653s user0m0.001s sys 0m0.003s real1m0.253s user0m0.001s sys 0m0.003s 8.3.7 - Solaris 10 8/07 s10x_u4wos_12b X86 real0m38.981s user0m0.002s sys 0m0.004s real0m39.879s user0m0.002s sys 0m0.004s real0m39.111s user0m0.002s sys 0m0.004s 8.4rc1 - Solaris 10 8/07 s10x_u4wos_12b X86 real0m50.647s user0m0.002s sys 0m0.004s real0m49.453s user0m0.002s sys 0m0.004s real0m49.725s user0m0.002s sys 0m0.004s Alan On Thu, Jun 18, 2009 at 3:39 PM, Tom Lane t...@sss.pgh.pa.us wrote: Kevin Grittner kevin.gritt...@wicourts.gov writes: Tom Lane t...@sss.pgh.pa.us wrote: I am unable to duplicate any slowdown on this test case. [ Kevin can ] It'd be useful first off to figure out if it's a CPU or I/O issue. Is there any visible difference in vmstat output? Also, try turning off autovacuum in both cases, just to see if that's related. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] PSQLException: FATAL: semctl(9335088, 3, SETVAL, 0) failed
Dear Sir/Madam I am running postgresql 8.1 on winows 2003 R2 in a production scenario. I am getting the following error from my java application while trying to connect to postgresql. org.postgresql.util.PSQLException: FATAL: semctl(9335088, 3, SETVAL, 0) failed: A non-blocking socket operation could not be completed immediately. Please list me the probable causes of the error. Rajdeep
Re: [HACKERS] 8.4 open item: copy performance regression?
Yes, you are right. I thought that they were absolute function counts. The data makes more sense now. Regards, Ken On Thu, Jun 18, 2009 at 07:03:34PM -0500, Kevin Grittner wrote: Kenneth Marshall k...@rice.edu wrote: What is not clear from Stefen's function listing is how the 8.4 server could issue 33% more XLogInsert() and CopyReadLine() calls than the 8.3.7 server using the same input file. I thought those were profiling numbers -- the number of times a timer checked what was executing and found it in that method. Which suggests that those two methods are probably slower now than in 8.3.7, at least in some environments. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PSQLException: FATAL: semctl(9335088, 3, SETVAL, 0) failed
Rajdeep Das sendrajs...@gmail.com writes: I am running postgresql 8.1 on winows 2003 R2 in a production scenario. Don't do that. 8.1 for Windows was declared obsolete quite some time ago. We are only supporting 8.2 and up. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 open item: copy performance regression?
Kevin Grittner wrote: 8.3.7 real0m24.249s real0m24.054s real0m24.361s 8.4rc1 real0m33.503s real0m34.198s real0m33.931s Ugh. This looks like a poster child case for a benchfarm ... Is there any chance you guys could triangulate this a bit? Good initial triangulation points might be the end of each commitfest. (I have a vested interest in making sure COPY performance doesn't regress, since it will affect parallel restore's speed in spades.) cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 open item: copy performance regression?
On 6/19/09, Andrew Dunstan and...@dunslane.net wrote: Kevin Grittner wrote: 8.3.7 real0m24.249s real0m24.054s real0m24.361s 8.4rc1 real0m33.503s real0m34.198s real0m33.931s Ugh. This looks like a poster child case for a benchfarm ... Is there any chance you guys could triangulate this a bit? Good initial triangulation points might be the end of each commitfest. (I have a vested interest in making sure COPY performance doesn't regress, since it will affect parallel restore's speed in spades.) git bisect? -- marko -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 open item: copy performance regression?
Andrew Dunstan wrote: Kevin Grittner wrote: 8.3.7 real0m24.249s real0m24.054s real0m24.361s 8.4rc1 real0m33.503s real0m34.198s real0m33.931s Ugh. This looks like a poster child case for a benchfarm ... indeed... Is there any chance you guys could triangulate this a bit? Good initial triangulation points might be the end of each commitfest. (I have a vested interest in making sure COPY performance doesn't regress, since it will affect parallel restore's speed in spades.) Maybe parallel restore is the issue why we haven't noticed this earlier. The case that regressed this way is WAL logged COPY, COPY that can bypass WAL (which typically happens in parallel restore now) is actually a bit faster in my testing in 8.4. I will try and see if I can figure out what caused the regression... Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 open item: copy performance regression?
Just eyeing the code ... another thing we changed since 8.3 is to enable posix_fadvise() calls for WAL. Any of the complaints want to try diking out this bit of code (near line 2580 in src/backend/access/transam/xlog.c)? #if defined(USE_POSIX_FADVISE) defined(POSIX_FADV_DONTNEED) if (!XLogArchivingActive() (get_sync_bit(sync_method) PG_O_DIRECT) == 0) (void) posix_fadvise(openLogFile, 0, 0, POSIX_FADV_DONTNEED); #endif regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] rc1 tarball contains partially outdated/missing man pages
On Thursday 18 June 2009 22:48:53 Tom Lane wrote: Peter Eisentraut pete...@gmx.net writes: I noticed that the rc1 tarball does not contain man pages for CREATE/ALTER/DROP FOREIGN DATA WRAPPER/SERVER/USER MAPPING. Just eyeballing the files, I notice that those ref pages seem not to contain this boilerplate: manvolnum7/manvolnum which is mostly(?) present in others. Maybe the cause? Great find. -- Fixed. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] rc1 tarball contains partially outdated/missing man pages
On Thursday 18 June 2009 23:15:53 Tom Lane wrote: I wrote: One thing I notice is that the table and with entries are not coming out as intended. The file names are all caps: -rw-r--r-- 1 tglusers 18 Jun 12 01:37 WITH.7 -rw-r--r-- 1 tglusers 18 Jun 12 01:37 TABLE.7 and the content surely isn't what was meant: $ more TABLE.7 .so man7/SELECT.7 $ more WITH.7 .so man7/SELECT.7 The content is pretty much what was meant, except that the case is wrong. Typing man SELECT will find a lower case select.7 man page, but the .so mechanism apparently doesn't. The simple fix is to convert all man page names to upper case, which is the default in the tools anyway. BTW, as far as that particular point goes: maybe we could fix the tools issues underlying this, but I'm tempted to think that it's not worth the trouble, because making these man pages be aliases for SELECT is just the Wrong Thing anyway. I think we should just split them off and have them be documented as separate top-level reference pages. Umm, the reason why it is that way now is that you did not want them to be separate top-level man pages: http://archives.postgresql.org/message-id/19886.1226171...@sss.pgh.pa.us -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 open item: copy performance regression?
Tom Lane wrote: Just eyeing the code ... another thing we changed since 8.3 is to enable posix_fadvise() calls for WAL. Any of the complaints want to try diking out this bit of code (near line 2580 in src/backend/access/transam/xlog.c)? #if defined(USE_POSIX_FADVISE) defined(POSIX_FADV_DONTNEED) if (!XLogArchivingActive() (get_sync_bit(sync_method) PG_O_DIRECT) == 0) (void) posix_fadvise(openLogFile, 0, 0, POSIX_FADV_DONTNEED); #endif doesn't seem to cause any noticable difference for me... Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] rc1 tarball contains partially outdated/missing man pages
Peter Eisentraut pete...@gmx.net writes: On Thursday 18 June 2009 23:15:53 Tom Lane wrote: BTW, as far as that particular point goes: maybe we could fix the tools issues underlying this, but I'm tempted to think that it's not worth the trouble, because making these man pages be aliases for SELECT is just the Wrong Thing anyway. I think we should just split them off and have them be documented as separate top-level reference pages. Umm, the reason why it is that way now is that you did not want them to be separate top-level man pages: http://archives.postgresql.org/message-id/19886.1226171...@sss.pgh.pa.us Well, at the time I thought that WITH would only be a sub-clause of SELECT. The idea that we might allow it to be attached to INSERT etc causes me to revise my opinion a bit. Do you have a preference one way or the other about how to do this? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 open item: copy performance regression?
Tom Lane wrote: Just eyeing the code ... another thing we changed since 8.3 is to enable posix_fadvise() calls for WAL. Any of the complaints want to try diking out this bit of code (near line 2580 in src/backend/access/transam/xlog.c)? #if defined(USE_POSIX_FADVISE) defined(POSIX_FADV_DONTNEED) if (!XLogArchivingActive() (get_sync_bit(sync_method) PG_O_DIRECT) == 0) (void) posix_fadvise(openLogFile, 0, 0, POSIX_FADV_DONTNEED); #endif ok after a bit of bisecting I'm happy to announce the winner of the contest: http://archives.postgresql.org/pgsql-committers/2008-11/msg00054.php this patch causes a 25-30% performance regression for WAL logged copy, however in the WAL bypass case (maybe that was what got tested?) it results in a 20% performance increase. the raw numbers using the upthread posted minimal postgresql.conf are: post patch/wal logged: 4min10s/4min19/4min12 post patch/wal bypass: 1m55s/1m58s/2m00 prepatch/wal logged: 2m55s/3min00/2m59 prepatch/wal bypass: 2m22s/2m18s/2m20s Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 open item: copy performance regression?
On Fri, Jun 19, 2009 at 07:49:31PM +0200, Stefan Kaltenbrunner wrote: Tom Lane wrote: Just eyeing the code ... another thing we changed since 8.3 is to enable posix_fadvise() calls for WAL. Any of the complaints want to try diking out this bit of code (near line 2580 in src/backend/access/transam/xlog.c)? #if defined(USE_POSIX_FADVISE) defined(POSIX_FADV_DONTNEED) if (!XLogArchivingActive() (get_sync_bit(sync_method) PG_O_DIRECT) == 0) (void) posix_fadvise(openLogFile, 0, 0, POSIX_FADV_DONTNEED); #endif ok after a bit of bisecting I'm happy to announce the winner of the contest: http://archives.postgresql.org/pgsql-committers/2008-11/msg00054.php this patch causes a 25-30% performance regression for WAL logged copy, however in the WAL bypass case (maybe that was what got tested?) it results in a 20% performance increase. the raw numbers using the upthread posted minimal postgresql.conf are: post patch/wal logged: 4min10s/4min19/4min12 post patch/wal bypass: 1m55s/1m58s/2m00 prepatch/wal logged: 2m55s/3min00/2m59 prepatch/wal bypass: 2m22s/2m18s/2m20s Stefan Great! Maybe just increasing the size of the BULKWRITE ring, possibly as a function of the shared_memory is all that is needed. 256kB is the currently coded ring_size in storage/buffer/freelist.c Ken -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 open item: copy performance regression?
Stefan Kaltenbrunner ste...@kaltenbrunner.cc writes: ok after a bit of bisecting I'm happy to announce the winner of the contest: http://archives.postgresql.org/pgsql-committers/2008-11/msg00054.php this patch causes a 25-30% performance regression for WAL logged copy, however in the WAL bypass case (maybe that was what got tested?) it results in a 20% performance increase. Hmm. What that patch actually changes is that it prevents a bulk insert (ie COPY in) from trashing the entire shared-buffers arena. I think the reason for the WAL correlation is that once it's filled the ring buffer, creating new pages requires writing out old ones, and the WAL-before-data rule means that the copy process has to block waiting for WAL to go down to disk before it can write. When it's allowed to use the whole arena there is more chance for some of that writing to be done by the walwriter or bgwriter. But the details are going to depend on the platform's CPU vs I/O balance, which no doubt explains why some of us don't see it. I don't think we want to revert that patch --- not trashing the whole buffer arena seems like a Good Thing from a system-wide point of view, even if it makes individual COPY operations go slower. However, we could maybe play around with the tradeoffs a bit. In particular it seems like it would be useful to experiment with different ring buffer sizes. Could you try increasing the ring size allowed in src/backend/storage/buffer/freelist.c for the BULKWRITE case *** *** 384,389 --- 384,392 case BAS_BULKREAD: ring_size = 256 * 1024 / BLCKSZ; break; + case BAS_BULKWRITE: + ring_size = 256 * 1024 / BLCKSZ; + break; case BAS_VACUUM: ring_size = 256 * 1024 / BLCKSZ; break; and see if maybe we can buy back most of the loss with not too much of a ring size increase? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 open item: copy performance regression?
Tom Lane wrote: Stefan Kaltenbrunner ste...@kaltenbrunner.cc writes: ok after a bit of bisecting I'm happy to announce the winner of the contest: http://archives.postgresql.org/pgsql-committers/2008-11/msg00054.php this patch causes a 25-30% performance regression for WAL logged copy, however in the WAL bypass case (maybe that was what got tested?) it results in a 20% performance increase. Hmm. What that patch actually changes is that it prevents a bulk insert (ie COPY in) from trashing the entire shared-buffers arena. I think the reason for the WAL correlation is that once it's filled the ring buffer, creating new pages requires writing out old ones, and the WAL-before-data rule means that the copy process has to block waiting for WAL to go down to disk before it can write. When it's allowed to use the whole arena there is more chance for some of that writing to be done by the walwriter or bgwriter. But the details are going to depend on the platform's CPU vs I/O balance, which no doubt explains why some of us don't see it. hmm - In my case both the CPU (an Intel E5530 Nehalem) and the IO subsystem (8GB Fiberchannel connected NetApp with 4GB cache) are pretty fast. and even with say fsync=off 8.4RC1 is only slightly faster than 8.3 with the same config and fsync=on so maybe there is a secondary effect at play too. I don't think we want to revert that patch --- not trashing the whole buffer arena seems like a Good Thing from a system-wide point of view, even if it makes individual COPY operations go slower. However, we could maybe play around with the tradeoffs a bit. In particular it seems like it would be useful to experiment with different ring buffer sizes. Could you try increasing the ring size allowed in src/backend/storage/buffer/freelist.c for the BULKWRITE case *** *** 384,389 --- 384,392 case BAS_BULKREAD: ring_size = 256 * 1024 / BLCKSZ; break; + case BAS_BULKWRITE: + ring_size = 256 * 1024 / BLCKSZ; + break; case BAS_VACUUM: ring_size = 256 * 1024 / BLCKSZ; break; and see if maybe we can buy back most of the loss with not too much of a ring size increase? already started testing that once I found the offending commit. 256 * 1024 / BLCKSZ 4min10s/4min19/4min12 512 * 1024 / BLCKSZ 3min27s/3min32s 1024 * 1024 / BLCKSZ 3min14s/3min12s 2048 * 1024 / BLCKSZ 3min02/3min02 4096 * 1024 / BLCKSZ 2m59/2m58s 8192 * 1024 / BLCKSZ 2m59/2m59s so 4096 * 1024 / BLCKSZ seems to be the sweet spot and also results in more or less the same performance that 8.3 had. Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] rc1 tarball contains partially outdated/missing man pages
On Friday 19 June 2009 19:12:50 Tom Lane wrote: Well, at the time I thought that WITH would only be a sub-clause of SELECT. The idea that we might allow it to be attached to INSERT etc causes me to revise my opinion a bit. Do you have a preference one way or the other about how to do this? I can't really imagine it right now, but when INSERT actually supports this, I'm sure we can come up with something. At the moment I'm concentrating on tweaking it so that the current setup works in 8.4. Which I think it does now, but it would be good if others could give the man pages themselves and their installation some extra testing in rc2. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Suppressing occasional failures in copy2 regression test
On Mon, Jun 15, 2009 at 10:00 PM, David Fetter da...@fetter.org wrote:
On Mon, Jun 15, 2009 at 11:34:38AM -0400, Tom Lane wrote:
David Fetter da...@fetter.org writes:
* It's going to a lot of trouble to allow for the possibility of both
unordered results and of duplicate lines. If we disallow duplicate
lines in unordered result sets, we can get a big speed gain by using
hash-based comparisons.
Why not just sort the lines and compare?
Good point :)
Please find attached the updated script and test cases. Changes since last
submission:
.) Script uses tabs for indentation.
.) Script almost passes perlcritic.com at 'stern' level.
.) Correct some RE matches, so that a ? mark only at the beginning of the
line matches (^ anchor).
.) Employ hybrid approach to support RE in unordered set, and to get better
performance:
If there's no RE line in an unordered group of lines then perform sort
on both arrays and then compare. If there _is_ an RE line in unordered group
of lines, then do the O(n^2) processing to eliminate common lines and then
report on missing lines.
TODO:
.) Using Tie::File to make code a little cleaner.
I agree that the choice of the hybrid approach for Unordered Set comparison
makes script too indented, and maybe a little hard on eyes, but it's pretty
simple and I have tried to delineate the major sections with proper
comments.
Best regards,
--
Lets call it Postgres
EnterpriseDB http://www.enterprisedb.com
gurjeet[.sin...@enterprisedb.com
singh.gurj...@{ gmail | hotmail | indiatimes | yahoo }.com
Mail sent from my BlackLaptop device
neurodiff.pl
Description: Binary data
expected.out
Description: Binary data
result.out
Description: Binary data
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Suppressing occasional failures in copy2 regression test
About the comment in chomp: did you try to use different values of $/? -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 open item: copy performance regression?
so 4096 * 1024 / BLCKSZ seems to be the sweet spot and also results in more or less the same performance that 8.3 had. Can some folks test this with different size COPYs? That's both larger/smaller tables, and larger/smaller rows. We should also test copy with large blob data. -- Josh Berkus PostgreSQL Experts Inc. www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Suppressing occasional failures in copy2 regression test
On Sat, Jun 20, 2009 at 2:06 AM, Alvaro Herrera
alvhe...@commandprompt.comwrote:
About the comment in chomp: did you try to use different values of $/?
Nope.. I think my first mail in this thread declared me a noob in perl :).
So I'd appreciate if someone could improve it perl-wise.
Thanks and best regards,
--
Lets call it Postgres
EnterpriseDB http://www.enterprisedb.com
gurjeet[.sin...@enterprisedb.com
singh.gurj...@{ gmail | hotmail | indiatimes | yahoo }.com
Mail sent from my BlackLaptop device
Re: [HACKERS] concurrent COPY performance
On Wed, Jun 17, 2009 at 3:44 AM, Kevin Grittner kevin.gritt...@wicourts.gov
wrote:
Andrew Dunstan and...@dunslane.net wrote:
If a table is created or truncated in the same transaction that does
the load, and archiving is not on, the COPY is not WALed.
Slightly off topic, but possibly relevant to the overall process:
those are the same conditions under which I would love to see the
rows inserted with the hint bits showing successful commit and the
transaction ID showing frozen. We currently do a VACUUM FREEZE
ANALYZE after such a load, to avoid burdening random users with the
writes. It would be nice not to have to write all the pages again
right after a load.
+1 (if it is doable, that is).
Best regards,
--
Lets call it Postgres
EnterpriseDB http://www.enterprisedb.com
gurjeet[.sin...@enterprisedb.com
singh.gurj...@{ gmail | hotmail | indiatimes | yahoo }.com
Mail sent from my BlackLaptop device
[HACKERS] cassert: array size exceeds the maximum allowed (134217727)
I'm getting a failed assertion on 8.4b1. With cassert off it crashes the postmaster. The assertion is: ERROR: array size exceeds the maximum allowed (134217727) Here is the query: select * from (select generate_series(1,500) as id) c left outer join foo_status a on c.id = a.id; note: replacing generate_series with table still gives the error. create or replace view foo_status as select * from ( select foo_id as id, array_to_string(array_agg(jc.name), ', ') as jc_list from foo_session left join (select * from bat where finished is null) q using (foo_id) left join bar using (bar_id) left join baz jc using (baz_id) where stop is null group by id, session_start, adb_pid ) q; all the tables are quite small. foo status runs fine normally and returns in a couple of msec. merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] cassert: array size exceeds the maximum allowed (134217727)
On Fri, Jun 19, 2009 at 5:27 PM, Merlin Moncuremmonc...@gmail.com wrote: I'm getting a failed assertion on 8.4b1. With cassert off it crashes correction: I'm getting this on 8.4rc1 :-). merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] cassert: array size exceeds the maximum allowed (134217727)
Merlin Moncure mmonc...@gmail.com writes: I'm getting a failed assertion on 8.4b1. With cassert off it crashes the postmaster. The assertion is: ERROR: array size exceeds the maximum allowed (134217727) Could we have a complete example, not just part of one? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 open item: copy performance regression?
On Fri, 2009-06-19 at 14:11 -0400, Tom Lane wrote: Stefan Kaltenbrunner ste...@kaltenbrunner.cc writes: ok after a bit of bisecting I'm happy to announce the winner of the contest: http://archives.postgresql.org/pgsql-committers/2008-11/msg00054.php this patch causes a 25-30% performance regression for WAL logged copy, however in the WAL bypass case (maybe that was what got tested?) it results in a 20% performance increase. Hmm. What that patch actually changes is that it prevents a bulk insert (ie COPY in) from trashing the entire shared-buffers arena. I think the reason for the WAL correlation is that once it's filled the ring buffer, creating new pages requires writing out old ones, and the WAL-before-data rule means that the copy process has to block waiting for WAL to go down to disk before it can write. When it's allowed to use the whole arena there is more chance for some of that writing to be done by the walwriter or bgwriter. But the details are going to depend on the platform's CPU vs I/O balance, which no doubt explains why some of us don't see it. I don't think we want to revert that patch --- not trashing the whole buffer arena seems like a Good Thing from a system-wide point of view, even if it makes individual COPY operations go slower. However, we could maybe play around with the tradeoffs a bit. In particular it seems like it would be useful to experiment with different ring buffer sizes. Could you try increasing the ring size allowed in src/backend/storage/buffer/freelist.c for the BULKWRITE case Yes, that's definitely the cause. The ring buffer technique was prototyped in 8.3 and a ring of 32 blocks was found not to be sufficient, which was one reason we didn't try to commit that then. At the time, I also proposed a filled buffer list change to bufmgr to allow bgwriter to preferentially target COPY's filled blocks, which would also help with this effect. You'll get better benefit from decreasing wal_writer_delay also. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 8.4 open item: copy performance regression?
On Fri, 19 Jun 2009, Stefan Kaltenbrunner wrote: In my case both the CPU (an Intel E5530 Nehalem) and the IO subsystem (8GB Fiberchannel connected NetApp with 4GB cache) are pretty fast. The server Alan identified as Solaris 10 8/07 s10x_u4wos_12b X86 has a Xeon E5320 (1.86GHz) and a single boring SAS drive in it (Sun X4150). The filesystem involved in that particular case is UFS, which I am suspicious of as being part of why the problem is so pronounced there--the default UFS tuning is pretty lightweight in terms of how much caching it does. Not sure if Alan ran any tests against the big ZFS volume on the other sever, I think all the results he posted were from the UFS boot drive there too. so 4096 * 1024 / BLCKSZ seems to be the sweet spot and also results in more or less the same performance that 8.3 had. It looks like it's a touch higher on our 8/07 system, it levels out at 8192 * (haven't checked the other one yet). I'm seeing this, using Alan's original test set size (to make sure I was running exactly the same test) and just grabbing the low/high from a set of 3 runs: 8.3.7: 0m39.266s 0m43.269s (alan: 36.2 - 39.2) 256:0m50.435s 0m51.944s (alan: 48.1 - 50.6) 1024: 0m47.299s 0m49.346s 4096: 0m43.725s 0m46.116s 8192: 0m40.715s 0m42.480s 16384: 0m41.318s 0m42.118s 65536: 0m41.675s 0m42.955s I collected some iostat data here as well for some of the runs (the vmstat data was harder to read, this being Solaris, and didn't seem to add anything). I'm seeing lines like this with the default ring buffer of 256 *: ttysd1 sd2 nfs1 cpu tin tout kps tps serv kps tps serv kps tps serv us sy wt id 0 322 12 10 41371 275400 00 12 11 0 78 0 166 0 00 46246 338000 00 14 10 0 76 0 164 0 00 44874 306810 00 13 9 0 78 Obviously sd2 is where the database and source file are at. Basically, about one core (out of four) tied up with a pretty even split of user/system time. Using the highest ring size I tried, 65536 *, gives lines that look like this: ttysd1 sd2 nfs1 cpu tin tout kps tps serv kps tps serv kps tps serv us sy wt id 0 163 0 00 56696 429100 00 20 12 0 68 0 166 0 00 58554 454200 00 21 12 0 67 0 168 0 00 56057 430800 00 21 12 0 67 So it seems like increasing the ring size helps saturate the disks better, went from ~45MB/s to 57MB/s. What's kind of interesting is to compare this against the 8.3.7 run, which is the fastest of them all, which I was expecting to find had the highest write rate of them all; ttysd1 sd2 nfs1 cpu tin tout kps tps serv kps tps serv kps tps serv us sy wt id 0 83 0 00 47654 212100 00 23 8 0 69 0 240 0 00 44198 215010 00 19 8 0 73 0 83 0 00 37750 111010 00 21 6 0 72 That's actually doing less I/O per capita, which is why it's also got less waiting for I/O%, but it's completing the most work. This makes me wonder if in addition to the ring buffering issue, there isn't just plain more writing per average completed transaction in 8.4 with this type of COPY. This might explain why even with the expanded ring buffer, both Stephan and my test runs still showed a bit of a regression against 8.3. I'm guessing we have a second, smaller shooter here involved as well. In any case, a bump of the ring multiplier to either 4096 or 8192 eliminates the worst of the regression here, good improvement so far. -- * Greg Smith gsm...@gregsmith.com http://www.gregsmith.com Baltimore, MD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
