Re: [HACKERS] Create collation incorrect error code

2012-05-31 Thread Tom Lane
Magnus Hagander writes: > On Thu, May 31, 2012 at 8:16 PM, Tom Lane wrote: >> What platform? > Ubuntu 12.04. FWIW, I get the same on Fedora 16. It works if I say locale='nb_NO.utf8' to prevent case-folding, so apparently case of the locale name has something to do with it ...

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-05-31 Thread Simon Riggs
On 31 May 2012 19:09, Sergey Koposov wrote: > On Thu, 31 May 2012, Simon Riggs wrote: > >> >> That struck me as a safe and easy optimisation. This was a problem I'd >> been trying to optimise for 9.2, so I've written a patch that appears >> simple and clean enough to be applied directly. > > > Tha

Re: [HACKERS] Re: [COMMITTERS] pgsql: Send new protocol keepalive messages to standby servers.

2012-05-31 Thread Tom Lane
Robert Haas writes: > Now, measuring time skew is potentially a useful thing to do, if we > believe that this will actually give us an accurate measurement of > what the time skew is, because there are a whole series of things that > people want to do which involve subtracting a slave timestamp fr

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-31 Thread Jeff Janes
On Wed, May 30, 2012 at 6:10 PM, Sergey Koposov wrote: > On Wed, 30 May 2012, Jeff Janes wrote: > >> But anyway, is idt_match a fairly static table?  If so, I'd partition >> that into 16 tables, and then have each one of your tasks join against >> a different one of those tables.  That should reli

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-05-31 Thread Jeff Janes
On Thu, May 31, 2012 at 11:09 AM, Sergey Koposov wrote: > On Thu, 31 May 2012, Simon Riggs wrote: > >> >> That struck me as a safe and easy optimisation. This was a problem I'd >> been trying to optimise for 9.2, so I've written a patch that appears >> simple and clean enough to be applied directl

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-31 Thread Jeff Janes
On Thu, May 31, 2012 at 11:50 AM, Robert Haas wrote: > > This test case is unusual because it hits a whole series of buffers > very hard.  However, there are other cases where this happens on a > single buffer that is just very, very hot, like the root block of a > btree index, where the pin/unpin

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-31 Thread Jeff Janes
On Thu, May 31, 2012 at 9:17 AM, Robert Haas wrote: > > Oh, ho.  So from this we can see that the problem is that we're > getting huge amounts of spinlock contention when pinning and unpinning > index pages. > > It would be nice to have a self-contained reproducible test case for > this, so that w

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-31 Thread Jeff Janes
On Sun, May 27, 2012 at 11:45 AM, Sergey Koposov wrote: > Hi, > > I did another test using the same data and the same code, which I've > provided before and the performance of the single thread seems to be > degrading quadratically with the number of threads. > > Here are the results: > Nthreads T

Re: [HACKERS] pgbench--new transaction type

2012-05-31 Thread Jeff Janes
On Sun, Jun 19, 2011 at 3:30 PM, Greg Smith wrote: > I applied Jeff's patch but changed this to address concerns about the > program getting stuck running for too long in the function: > > #define plpgsql_loops   512 > > This would be better named as "plpgsql_batch_size" or something similar > ins

Re: [HACKERS] Figuring out shared buffer pressure

2012-05-31 Thread Daniel Farina
On Thu, May 31, 2012 at 10:11 AM, Bruce Momjian wrote: > However, this doesn't help people configure shared buffers larger (e.g. > 35%) if their working set is larger.  Right now, I don't see how a user > would know this is happening.  On the flip side, they might have a > smaller working set than

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-31 Thread Robert Haas
On Thu, May 31, 2012 at 3:25 PM, Merlin Moncure wrote: > Hm, couple questions: how do you determine if/when to un-nail a > buffer, and who makes that decision (bgwriter?) Well, I think some experimentation might be required, but my first thought is to tie it into buffer eviction. If we record a

Re: [HACKERS] [RFC] Interface of Row Level Security

2012-05-31 Thread Kohei KaiGai
2012/5/31 Kohei KaiGai : > 2012/5/31 Robert Haas : >>> If we would have an "ideal optimizer", I'd still like the optimizer to >>> wipe out redundant clauses transparently, rather than RLSBYPASS >>> permissions, because it just controls all-or-nothing stuff. >>> For example, if tuples are categorize

Re: [HACKERS] [RFC] Interface of Row Level Security

2012-05-31 Thread Kohei KaiGai
2012/5/31 Robert Haas : >> If we would have an "ideal optimizer", I'd still like the optimizer to >> wipe out redundant clauses transparently, rather than RLSBYPASS >> permissions, because it just controls all-or-nothing stuff. >> For example, if tuples are categorized to unclassified, classified o

Re: [HACKERS] Re: [COMMITTERS] pgsql: Send new protocol keepalive messages to standby servers.

2012-05-31 Thread Robert Haas
On Wed, May 30, 2012 at 12:17 PM, Fujii Masao wrote: > OTOH, I wonder whether we really need to send keepalive messages > periodically to calculate a network latency. ISTM we don't unless a network > latency varies from situation to situation so frequently and we'd like to > monitor that in almost

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-31 Thread Merlin Moncure
On Thu, May 31, 2012 at 1:50 PM, Robert Haas wrote: > On Thu, May 31, 2012 at 2:03 PM, Merlin Moncure wrote: >> On Thu, May 31, 2012 at 11:54 AM, Sergey Koposov >> wrote: >>> On Thu, 31 May 2012, Robert Haas wrote: >>> Oh, ho.  So from this we can see that the problem is that we're ge

Re: [HACKERS] extending relations more efficiently

2012-05-31 Thread Jeff Davis
On Tue, 2012-05-01 at 10:08 -0400, Robert Haas wrote: > We've previously discussed the possible desirability of extending > relations in larger increments, rather than one block at a time, for > performance reasons. I attempted to determine how much performance we > could possibly buy this way, an

Re: [HACKERS] \conninfo and SSL

2012-05-31 Thread Magnus Hagander
On Thu, May 31, 2012 at 9:04 PM, Bruce Momjian wrote: > On startup, psql shows the SSL information: > >        $ psql 'sslmode=require host=localhost' >        psql (9.2beta1) >        SSL connection (cipher: DHE-RSA-AES256-SHA, bits: 256) > > However, \conninfo does not mention SSL: > >        po

[HACKERS] \conninfo and SSL

2012-05-31 Thread Bruce Momjian
On startup, psql shows the SSL information: $ psql 'sslmode=require host=localhost' psql (9.2beta1) SSL connection (cipher: DHE-RSA-AES256-SHA, bits: 256) However, \conninfo does not mention SSL: postgres=> \conninfo You are connected to database "postgres

Re: [HACKERS] Issues with MinGW W64

2012-05-31 Thread Robert Haas
On Thu, May 31, 2012 at 2:13 PM, Tom Lane wrote: > Robert Haas writes: >> On Thu, May 31, 2012 at 8:15 AM, Johann 'Myrkraverk' Oskarsson >> wrote: >>> Is this something to discuss with the MinGW W64 team? > >> My viewpoint on this (which is different than Tom's) is that we're >> probably not ent

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-31 Thread Robert Haas
On Thu, May 31, 2012 at 2:03 PM, Merlin Moncure wrote: > On Thu, May 31, 2012 at 11:54 AM, Sergey Koposov > wrote: >> On Thu, 31 May 2012, Robert Haas wrote: >> >>> Oh, ho.  So from this we can see that the problem is that we're >>> getting huge amounts of spinlock contention when pinning and un

Re: [HACKERS] Figuring out shared buffer pressure

2012-05-31 Thread Cédric Villemain
Le jeudi 31 mai 2012 19:11:07, Bruce Momjian a écrit : > On Wed, May 30, 2012 at 05:55:07PM -0400, Bruce Momjian wrote: > > > > Seems buffers_alloc is the number of calls to StrategyGetBuffer(), > > > > which tells how many time we have requested a buffer. Not sure how > > > > that helps measure b

Re: [HACKERS] Create collation incorrect error code

2012-05-31 Thread Magnus Hagander
On Thu, May 31, 2012 at 8:16 PM, Tom Lane wrote: > Magnus Hagander writes: >> template1=# create collation "nb_NO.utf8" (locale=nb_NO.utf8); >> ERROR:  could not create locale "nb_no.utf8": Success > > What platform? D'uh, sorry. Ubuntu 12.04. --  Magnus Hagander  Me: http://www.hagander.net

Re: [HACKERS] Create collation incorrect error code

2012-05-31 Thread Tom Lane
Magnus Hagander writes: > template1=# create collation "nb_NO.utf8" (locale=nb_NO.utf8); > ERROR: could not create locale "nb_no.utf8": Success What platform? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to you

Re: [HACKERS] Issues with MinGW W64

2012-05-31 Thread Tom Lane
Robert Haas writes: > On Thu, May 31, 2012 at 8:15 AM, Johann 'Myrkraverk' Oskarsson > wrote: >> Is this something to discuss with the MinGW W64 team? > My viewpoint on this (which is different than Tom's) is that we're > probably not entitled to assume anything about what the system header > fi

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-05-31 Thread Sergey Koposov
On Thu, 31 May 2012, Simon Riggs wrote: That struck me as a safe and easy optimisation. This was a problem I'd been trying to optimise for 9.2, so I've written a patch that appears simple and clean enough to be applied directly. Thanks! The patch indeed improved the timings, The dropping of

[HACKERS] Create collation incorrect error code

2012-05-31 Thread Magnus Hagander
Strangeness: template1=# create collation "nb_NO.utf8" (locale=nb_NO.utf8); ERROR: could not create locale "nb_no.utf8": Success Clearly it's not successful... On some runs, I get: template1=# create collation "nb_NO.utf8" (locale=nb_NO.utf8); ERROR: could not create locale "nb_no.utf8": No s

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-31 Thread Merlin Moncure
On Thu, May 31, 2012 at 11:54 AM, Sergey Koposov wrote: > On Thu, 31 May 2012, Robert Haas wrote: > >> Oh, ho.  So from this we can see that the problem is that we're >> getting huge amounts of spinlock contention when pinning and unpinning >> index pages. >> >> It would be nice to have a self-con

Re: [HACKERS] heap_form_tuple crashing

2012-05-31 Thread Atri Sharma
On Thu, May 31, 2012 at 7:01 PM, Tom Lane wrote: > Atri Sharma writes: >> My call to heap_form_tuple looks like: >> val1=0; >> tupledesc1=BlessTupleDesc(node->ss.ss_currentRelation->rd_att); >> tuple=heap_form_tuple(tupledesc1,p1,&val1); > >> p1 is a pointer to a Datum instance which is created f

Re: [HACKERS] hot standby PSQL 9.1 Windows 2008 Servers

2012-05-31 Thread chinnaobi
Sorry to mention, In my setup the primary and standby servers receive same traffic, so no issue with the "network fault between the primary and the standby, but not between the primary and some of the clients updating it" -- View this message in context: http://postgresql.1045698.n5.nabble.com/R

Re: [HACKERS] hot standby PSQL 9.1 Windows 2008 Servers

2012-05-31 Thread chinnaobi
Dear Kevin, Thank you for your reply. Yeah I am writing an application using powershell, it's true it is not trivial and especially a guy like me who has no idea on database. You raised all the cases which I am muddling with, But currently I am testing this setup: Always standby server is confi

Re: [HACKERS] Figuring out shared buffer pressure

2012-05-31 Thread Bruce Momjian
On Wed, May 30, 2012 at 05:55:07PM -0400, Bruce Momjian wrote: > > > Seems buffers_alloc is the number of calls to StrategyGetBuffer(), which > > > tells how many time we have requested a buffer.  Not sure how that helps > > > measure buffer pressure. > > > > Once the linked list is empty, every r

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-31 Thread Sergey Koposov
On Thu, 31 May 2012, Robert Haas wrote: Oh, ho. So from this we can see that the problem is that we're getting huge amounts of spinlock contention when pinning and unpinning index pages. It would be nice to have a self-contained reproducible test case for this, so that we could experiment with

Re: [HACKERS] hot standby PSQL 9.1 Windows 2008 Servers

2012-05-31 Thread Kevin Grittner
chinnaobi wrote: > You mean when the primary which is going to switch its role to > standby might not have sent all the WAL records to the standby and > If it is switched to standby it has more WAL records than the > standby which is now serves as primary. Is it ?? What happens when there is a

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-31 Thread Merlin Moncure
On Thu, May 31, 2012 at 10:23 AM, Sergey Koposov wrote: > On Thu, 31 May 2012, Robert Haas wrote: >> >> >> Thanks.  How did you generate this perf report?  It's cool, because I >> haven't figured out how to make perf generate a report that is easily >> email-able, and it seems you have. > > > I di

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-31 Thread Robert Haas
On Thu, May 31, 2012 at 11:23 AM, Sergey Koposov wrote: > On Thu, 31 May 2012, Robert Haas wrote: >> >> Thanks.  How did you generate this perf report?  It's cool, because I >> haven't figured out how to make perf generate a report that is easily >> email-able, and it seems you have. > > I did pre

Re: [HACKERS] Draft release notes complete

2012-05-31 Thread Magnus Hagander
On Thu, May 31, 2012 at 5:55 PM, Bruce Momjian wrote: > On Tue, May 15, 2012 at 12:57:37PM -0400, Magnus Hagander wrote: >> On Fri, May 11, 2012 at 11:44 AM, Andrew Dunstan wrote: >> > >> > >> > On 05/11/2012 05:32 AM, Magnus Hagander wrote: >> >> >> >> >> >> But in the interest of actually being

Re: [HACKERS] Draft release notes complete

2012-05-31 Thread Bruce Momjian
On Tue, May 15, 2012 at 12:57:37PM -0400, Magnus Hagander wrote: > On Fri, May 11, 2012 at 11:44 AM, Andrew Dunstan wrote: > > > > > > On 05/11/2012 05:32 AM, Magnus Hagander wrote: > >> > >> > >> But in the interest of actually being productive - what *is* the > >> usecase for needing a 5 minute

Re: [HACKERS] Uh, I change my mind about commit_delay + commit_siblings (sort of)

2012-05-31 Thread Peter Geoghegan
On 31 May 2012 16:23, Tom Lane wrote: > Simon Riggs writes: >> In what way is it possibly destabilising? > > I'm prepared to believe that it only affects performance, but it could > be destabilizing to that.  It needs proper review and testing, and the > next CF is the right environment for that

Re: [HACKERS] hash index concurrency

2012-05-31 Thread David Fetter
On Wed, May 30, 2012 at 12:21:33AM -0400, Tom Lane wrote: > Robert Haas writes: > > On Tue, May 29, 2012 at 11:21 PM, Jeff Janes wrote: > >> 2) Only support bitmap scans and not ordinary tid scans (the way gin > >> indexes already do). > > > -1 on losing amgettuple. I regret that we lost that f

Re: [HACKERS] Uh, I change my mind about commit_delay + commit_siblings (sort of)

2012-05-31 Thread Peter Geoghegan
On 31 May 2012 16:26, Peter Geoghegan wrote: > On 31 May 2012 16:23, Tom Lane wrote: >> Simon Riggs writes: >>> In what way is it possibly destabilising? >> >> I'm prepared to believe that it only affects performance, but it could >> be destabilizing to that.  It needs proper review and testing,

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-31 Thread Claudio Freire
On Thu, May 31, 2012 at 12:25 PM, Tom Lane wrote: >> No, Tatsuo's patch attacks a phase dominated by latency in some >> setups. > > No, it does not.  The reason it's a win is that it avoids the O(N^2) > behavior in the server.  Whether the bandwidth savings is worth worrying > about cannot be prov

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-31 Thread Tom Lane
Claudio Freire writes: > On Thu, May 31, 2012 at 11:50 AM, Tom Lane wrote: >> The performance patches we applied to pg_dump over the past couple weeks >> were meant to relieve pain in situations where the big server-side >> lossage wasn't the dominant factor in runtime (ie, partial dumps). >> But

Re: [HACKERS] Uh, I change my mind about commit_delay + commit_siblings (sort of)

2012-05-31 Thread Peter Geoghegan
On 31 May 2012 14:58, Robert Haas wrote: > Fixing regressions before release is essential; improving performance > is not - especially when the improvement relates to a little-used > feature that you were proposing to get rid of two weeks ago. Yes, the fact that I wanted to get rid of commit_dela

Re: [HACKERS] Uh, I change my mind about commit_delay + commit_siblings (sort of)

2012-05-31 Thread Tom Lane
Simon Riggs writes: > On 31 May 2012 15:00, Tom Lane wrote: >> If we want to finish the beta cycle in a reasonable time period and get >> back to actual development, we have to refrain from adding more >> possibly-destabilizing development work to 9.2.  And that is what >> this is. > In what way

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-31 Thread Bruce Momjian
On Thu, May 31, 2012 at 11:04:12AM -0400, Robert Haas wrote: > On Thu, May 31, 2012 at 10:50 AM, Tom Lane wrote: > > Robert Haas writes: > >> On Thu, May 31, 2012 at 10:31 AM, Tom Lane wrote: > >>> I'm not; Jeff Janes is.  But you shouldn't be holding your breath > >>> anyway, since it's 9.3 mat

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-31 Thread Robert Haas
On Thu, May 31, 2012 at 10:50 AM, Tom Lane wrote: > Robert Haas writes: >> On Thu, May 31, 2012 at 10:31 AM, Tom Lane wrote: >>> I'm not; Jeff Janes is.  But you shouldn't be holding your breath >>> anyway, since it's 9.3 material at this point. > >> I agree we can't back-patch that change, but

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-31 Thread Claudio Freire
On Thu, May 31, 2012 at 11:50 AM, Tom Lane wrote: > The performance patches we applied to pg_dump over the past couple weeks > were meant to relieve pain in situations where the big server-side > lossage wasn't the dominant factor in runtime (ie, partial dumps). > But this one is targeting exactly

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-31 Thread Bruce Momjian
On Thu, May 31, 2012 at 10:50:51AM -0400, Tom Lane wrote: > Robert Haas writes: > > On Thu, May 31, 2012 at 10:31 AM, Tom Lane wrote: > >> I'm not; Jeff Janes is. �But you shouldn't be holding your breath > >> anyway, since it's 9.3 material at this point. > > > I agree we can't back-patch that

Re: [HACKERS] Uh, I change my mind about commit_delay + commit_siblings (sort of)

2012-05-31 Thread Simon Riggs
On 31 May 2012 15:00, Tom Lane wrote: > Simon Riggs writes: >> On 31 May 2012 13:16, Robert Haas wrote: >>> Frankly, I think this whole thing should be pushed to 9.3. > >> What matters is that we have a patch that provides a massive >> performance gain in write performance in just a few lines of

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-31 Thread Tom Lane
Robert Haas writes: > On Thu, May 31, 2012 at 10:31 AM, Tom Lane wrote: >> I'm not; Jeff Janes is.  But you shouldn't be holding your breath >> anyway, since it's 9.3 material at this point. > I agree we can't back-patch that change, but then I think we ought to > consider back-patching some var

Re: [HACKERS] [RFC] Interface of Row Level Security

2012-05-31 Thread Robert Haas
On Wed, May 30, 2012 at 3:26 PM, Kohei KaiGai wrote: > My preference is RLSBYPASS permission rather than the approach > with functions that return policy clause at run-time, because it needs > to invalidate prepared statement at random timing. > In case of this function approach, the RLS policy sh

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-31 Thread Robert Haas
On Thu, May 31, 2012 at 10:31 AM, Tom Lane wrote: > Claudio Freire writes: >> It's not clear whether Tom is already working on that O(N^2) fix in locking. > > I'm not; Jeff Janes is.  But you shouldn't be holding your breath > anyway, since it's 9.3 material at this point. I agree we can't back-

Re: [HACKERS] Issues with MinGW W64

2012-05-31 Thread Robert Haas
On Thu, May 31, 2012 at 8:15 AM, Johann 'Myrkraverk' Oskarsson wrote: MinGW W64's sys/stat.h #defines stat to be _stati64 and there is subsequently a compilation error in port.h: note: expected 'struct _stati64 *' but argument is of type 'struct stat *' error: conflicting

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-31 Thread Tom Lane
Claudio Freire writes: > It's not clear whether Tom is already working on that O(N^2) fix in locking. I'm not; Jeff Janes is. But you shouldn't be holding your breath anyway, since it's 9.3 material at this point. regards, tom lane -- Sent via pgsql-hackers mailing lis

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-31 Thread Claudio Freire
On Thu, May 31, 2012 at 11:17 AM, Robert Klemme wrote: > > OK, my fault was to assume you wanted to measure only your part, while > apparently you meant overall savings.  But Tom had asked for separate > measurements if I understood him correctly.  Also, that measurement of > your change would go

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-31 Thread Robert Haas
On Thu, May 31, 2012 at 7:31 AM, Sergey Koposov wrote: > On Wed, 30 May 2012, Robert Haas wrote: > >> I'd really like to find out exactly where all those s_lock calls are >> coming from.  Is there any way you can get oprofile to output a >> partial stack backtrace?  If you have perf it's very easy

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-31 Thread Robert Klemme
On Thu, May 31, 2012 at 4:07 PM, Tatsuo Ishii wrote: >> On Thu, May 31, 2012 at 10:45 AM, Tatsuo Ishii wrote: >>> Just for record, I rerun the test again with my single-LOCK patch, and >>> now total runtime of pg_dump is 113 minutes. >>> 188 minutes(9.0)->125 minutes(git master)->113 minutes(with

Re: [HACKERS] FailedAssertion("!(PrivateRefCount[i] == 0)", File: "bufmgr.c", Line: 1741

2012-05-31 Thread Robert Haas
On Thu, May 31, 2012 at 9:51 AM, Tom Lane wrote: > Robert Haas writes: >> The one thing that still seems a little odd to me is that this caused >> a pin count to get orphaned.  It seems reasonable that ignoring the >> AccessExclusiveLock could result in not-found errors trying to open a >> missin

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-31 Thread Tatsuo Ishii
> On Thu, May 31, 2012 at 10:45 AM, Tatsuo Ishii wrote: >> Just for record, I rerun the test again with my single-LOCK patch, and >> now total runtime of pg_dump is 113 minutes. >> 188 minutes(9.0)->125 minutes(git master)->113 minutes(with my patch). >> >> So far, I'm glad to see 40% time savings

Re: [HACKERS] Uh, I change my mind about commit_delay + commit_siblings (sort of)

2012-05-31 Thread Tom Lane
Simon Riggs writes: > On 31 May 2012 13:16, Robert Haas wrote: >> Frankly, I think this whole thing should be pushed to 9.3. > What matters is that we have a patch that provides a massive > performance gain in write performance in just a few lines of code, and > that should be committed to 9.2.

Re: [HACKERS] Uh, I change my mind about commit_delay + commit_siblings (sort of)

2012-05-31 Thread Robert Haas
On Thu, May 31, 2012 at 8:38 AM, Peter Geoghegan wrote: > On 31 May 2012 13:16, Robert Haas wrote: >> On Thu, May 31, 2012 at 6:19 AM, Simon Riggs wrote: >> Frankly, I think this whole thing should be pushed to 9.3.  The >> commit_delay and commit_siblings knobs suck, but they've sucked for a >>

Re: [HACKERS] FailedAssertion("!(PrivateRefCount[i] == 0)", File: "bufmgr.c", Line: 1741

2012-05-31 Thread Tom Lane
Robert Haas writes: > The one thing that still seems a little odd to me is that this caused > a pin count to get orphaned. It seems reasonable that ignoring the > AccessExclusiveLock could result in not-found errors trying to open a > missing relation, and even fsync requests on a missing relatio

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-31 Thread Robert Klemme
On Thu, May 31, 2012 at 10:45 AM, Tatsuo Ishii wrote: > Just for record, I rerun the test again with my single-LOCK patch, and > now total runtime of pg_dump is 113 minutes. > 188 minutes(9.0)->125 minutes(git master)->113 minutes(with my patch). > > So far, I'm glad to see 40% time savings at thi

Re: [HACKERS] heap_form_tuple crashing

2012-05-31 Thread Tom Lane
Atri Sharma writes: > My call to heap_form_tuple looks like: > val1=0; > tupledesc1=BlessTupleDesc(node->ss.ss_currentRelation->rd_att); > tuple=heap_form_tuple(tupledesc1,p1,&val1); > p1 is a pointer to a Datum instance which is created from a char array. Does that actually match the tupdesc yo

Re: [HACKERS] Uh, I change my mind about commit_delay + commit_siblings (sort of)

2012-05-31 Thread Peter Geoghegan
On 31 May 2012 13:16, Robert Haas wrote: > On Thu, May 31, 2012 at 6:19 AM, Simon Riggs wrote: > Frankly, I think this whole thing should be pushed to 9.3.  The > commit_delay and commit_siblings knobs suck, but they've sucked for a > long time, and it won't kill anybody to wait another release c

Re: [HACKERS] Uh, I change my mind about commit_delay + commit_siblings (sort of)

2012-05-31 Thread Simon Riggs
On 31 May 2012 13:16, Robert Haas wrote: > On Thu, May 31, 2012 at 6:19 AM, Simon Riggs wrote: >> I've looked at this more closely now and I can see that the call to >> XLogFlush() that is made from xact_redo_commit_internal() doesn't ever >> actually flush WAL, so whether we delay or not is comp

Re: [HACKERS] Issues with MinGW W64

2012-05-31 Thread Johann 'Myrkraverk' Oskarsson
Tom Lane writes: > Robert Haas writes: >> On Tue, May 29, 2012 at 9:04 AM, Johann 'Myrkraverk' Oskarsson >> wrote: >>> The header file crtdefs.h in MinGW typedefs errcode which conflicts >>> with Postgres' elog.h. > >> Eep. Maybe this is not directly relevant, but I guess my first >> question

Re: [HACKERS] Uh, I change my mind about commit_delay + commit_siblings (sort of)

2012-05-31 Thread Robert Haas
On Thu, May 31, 2012 at 6:19 AM, Simon Riggs wrote: > I've looked at this more closely now and I can see that the call to > XLogFlush() that is made from xact_redo_commit_internal() doesn't ever > actually flush WAL, so whether we delay or not is completely > irrelevant. > > So un-agreed. No chang

Re: [HACKERS] FailedAssertion("!(PrivateRefCount[i] == 0)", File: "bufmgr.c", Line: 1741

2012-05-31 Thread Robert Haas
On Thu, May 31, 2012 at 7:49 AM, Erik Rijkers wrote: > On Thu, May 31, 2012 13:14, Robert Haas wrote: >> On Thu, May 31, 2012 at 1:06 AM, Erik Rijkers wrote: >>> In my test, I run the bash code (the bits that I posted earlier) in a while >>> loop so that the >>> table >>> is CREATEd, COPYied int

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-05-31 Thread Simon Riggs
On 30 May 2012 12:10, Heikki Linnakangas wrote: > Hmm, we do this in smgrDoPendingDeletes: > > for (i = 0; i <= MAX_FORKNUM; i++) > { >        smgrdounlink(srel, i, false); > } > > So we drop the buffers for each relation fork separately, which means that > we scan the buffer pool four times. Rel

Re: [HACKERS] FailedAssertion("!(PrivateRefCount[i] == 0)", File: "bufmgr.c", Line: 1741

2012-05-31 Thread Erik Rijkers
On Thu, May 31, 2012 13:14, Robert Haas wrote: > On Thu, May 31, 2012 at 1:06 AM, Erik Rijkers wrote: >> In my test, I run the bash code (the bits that I posted earlier) in a while >> loop so that the >> table >> is CREATEd, COPYied into, and DROPped every few seconds -- perhaps that >> wasn't c

Re: [HACKERS] FailedAssertion("!(PrivateRefCount[i] == 0)", File: "bufmgr.c", Line: 1741

2012-05-31 Thread Simon Riggs
On 31 May 2012 12:14, Robert Haas wrote: > On Thu, May 31, 2012 at 1:06 AM, Erik Rijkers wrote: >> In my test, I run the bash code (the bits that I posted earlier) in a while >> loop so that the table >> is CREATEd, COPYied into, and DROPped every few seconds -- perhaps that >> wasn't clear.  T

Re: [HACKERS] heap_form_tuple crashing

2012-05-31 Thread Atri Sharma
On Thu, May 31, 2012 at 5:14 PM, Heikki Linnakangas wrote: > On 31.05.2012 14:42, Atri Sharma wrote: >> >> Another thing I wanted to ask was that would you recommend building >> tuples from strings directly or converting them to Datum first and >> then build the tuples from Datum instances? > > >

Re: [HACKERS] heap_form_tuple crashing

2012-05-31 Thread Heikki Linnakangas
On 31.05.2012 14:42, Atri Sharma wrote: Another thing I wanted to ask was that would you recommend building tuples from strings directly or converting them to Datum first and then build the tuples from Datum instances? It depends. If you have all the values in strings already, then BuildTupleF

Re: [HACKERS] heap_form_tuple crashing

2012-05-31 Thread Atri Sharma
On Thu, May 31, 2012 at 5:02 PM, Heikki Linnakangas wrote: > On 31.05.2012 13:42, Atri Sharma wrote: >> >> I am trying to call heap_form_tuple to create a tuple from a datum. >> >> My call to heap_form_tuple looks like: >> >> >> val1=0; >> tupledesc1=BlessTupleDesc(node->ss.ss_currentRelation->rd_

Re: [HACKERS] heap_form_tuple crashing

2012-05-31 Thread Heikki Linnakangas
On 31.05.2012 13:42, Atri Sharma wrote: I am trying to call heap_form_tuple to create a tuple from a datum. My call to heap_form_tuple looks like: val1=0; tupledesc1=BlessTupleDesc(node->ss.ss_currentRelation->rd_att); tuple=heap_form_tuple(tupledesc1,p1,&val1); p1 is a pointer to a Datum in

Re: [HACKERS] Uh, I change my mind about commit_delay + commit_siblings (sort of)

2012-05-31 Thread Peter Geoghegan
On 31 May 2012 11:19, Simon Riggs wrote: > I've looked at this more closely now and I can see that the call to > XLogFlush() that is made from xact_redo_commit_internal() doesn't ever > actually flush WAL, so whether we delay or not is completely > irrelevant. > > So un-agreed. No change required

Re: [HACKERS] FailedAssertion("!(PrivateRefCount[i] == 0)", File: "bufmgr.c", Line: 1741

2012-05-31 Thread Robert Haas
On Thu, May 31, 2012 at 1:06 AM, Erik Rijkers wrote: > In my test, I run the bash code (the bits that I posted earlier) in a while > loop so that the table > is CREATEd, COPYied into, and DROPped every few seconds -- perhaps that > wasn't clear.  That loop > is necessary; a few iterations are of

Re: [HACKERS] patch: avoid heavyweight locking on hash metapage

2012-05-31 Thread Robert Haas
On Thu, May 31, 2012 at 7:07 AM, Dickson S. Guedes wrote: > 2012/5/30 Robert Haas : >> I tested the effect of this by setting up a series of 5-minute >> read-only pgbench run at scale factor 300 with 8GB of shared buffers >> on the IBM POWER7 machine. > > I know it doesn't matter, but out of curio

Re: [HACKERS] patch: avoid heavyweight locking on hash metapage

2012-05-31 Thread Dickson S. Guedes
2012/5/30 Robert Haas : > I tested the effect of this by setting up a series of 5-minute > read-only pgbench run at scale factor 300 with 8GB of shared buffers > on the IBM POWER7 machine. I know it doesn't matter, but out of curiosity what OS you used? best regards, -- Dickson S. Guedes mail/xm

[HACKERS] heap_form_tuple crashing

2012-05-31 Thread Atri Sharma
Hi all, I am trying to call heap_form_tuple to create a tuple from a datum. My call to heap_form_tuple looks like: val1=0; tupledesc1=BlessTupleDesc(node->ss.ss_currentRelation->rd_att); tuple=heap_form_tuple(tupledesc1,p1,&val1); p1 is a pointer to a Datum instance which is created from a ch

Re: [HACKERS] GiST buffering build, bug in levelStep calculation

2012-05-31 Thread Alexander Korotkov
On Thu, May 31, 2012 at 1:36 PM, Heikki Linnakangas < heikki.linnakan...@enterprisedb.com> wrote: > I note that the calculations assume that leaf tuples and internal tuples > have similar sizes. We calculate the average leaf tuple size, and use that > to calculate the fan-out of internal pages. On

Re: [HACKERS] Uh, I change my mind about commit_delay + commit_siblings (sort of)

2012-05-31 Thread Simon Riggs
On 30 May 2012 17:19, Peter Geoghegan wrote: > On 30 May 2012 15:25, Simon Riggs wrote: >>> 1. It seems wrong to do it in xact_redo_commit_internal().  It won't >>> matter if commit_siblings>0 since there won't be any other backends >>> with transaction IDs anyway, but if commit_siblings==0 then

Re: [HACKERS] GiST buffering build, bug in levelStep calculation

2012-05-31 Thread Heikki Linnakangas
On 29.05.2012 23:46, Alexander Korotkov wrote: On Wed, May 30, 2012 at 12:25 AM, Tom Lane wrote: Alexander Korotkov writes: On Tue, May 29, 2012 at 11:42 PM, Tom Lane wrote: While I'm looking at this, is the first test involving effective_cache_size bulletproof either? In particular, is a

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-31 Thread Tatsuo Ishii
>>> We recently fixed a couple of O(N^2) loops in pg_dump, but those covered >>> extremely specific cases that might or might not have anything to do >>> with what you're seeing. The complainant was extremely helpful about >>> tracking down the problems: >>> http://archives.postgresql.org/pgsql-ge

Re: [HACKERS] [RFC] Interface of Row Level Security

2012-05-31 Thread Kohei KaiGai
2012/5/31 Yeb Havinga : > On 2012-05-30 21:26, Kohei KaiGai wrote: >> >> If we would have an "ideal optimizer", I'd still like the optimizer to >> wipe out redundant clauses transparently, rather than RLSBYPASS >> permissions, because it just controls all-or-nothing stuff. >> For example, if tuples

Re: [HACKERS] FailedAssertion("!(PrivateRefCount[i] == 0)", File: "bufmgr.c", Line: 1741

2012-05-31 Thread Heikki Linnakangas
On 31.05.2012 08:06, Erik Rijkers wrote: On Thu, May 31, 2012 03:30, Robert Haas wrote: On Wed, May 30, 2012 at 6:00 PM, Erik Rijkers wrote: directory 2012-05-30 23:40:57.909 CEST 3909 CONTEXT: writing block 5152 of relation base/21268/26569 xlog redo multi-insert (init): rel 1663/21

Re: [HACKERS] [RFC] Interface of Row Level Security

2012-05-31 Thread Yeb Havinga
On 2012-05-30 21:26, Kohei KaiGai wrote: If we would have an "ideal optimizer", I'd still like the optimizer to wipe out redundant clauses transparently, rather than RLSBYPASS permissions, because it just controls all-or-nothing stuff. For example, if tuples are categorized to unclassified, class