Re: [HACKERS] new group commit behavior not helping?

2012-03-31 Thread Jeff Janes
On Saturday, March 31, 2012, Jeff Janes wrote: > On Saturday, March 31, 2012, Robert Haas wrote: >> On Sun, Apr 1, 2012 at 1:40 AM, Jeff Janes wrote: > >>> It looks like in your case tps was still scaling with clients when you gave >>> up, so clients was probably too small. >> >> What is kind of

Re: [HACKERS] new group commit behavior not helping?

2012-03-31 Thread Robert Haas
On Sun, Apr 1, 2012 at 1:40 AM, Jeff Janes wrote: > On Saturday, March 31, 2012, Robert Haas wrote: >> Hoping to demonstrate the wonders of our new group commit code, I ran >> some benchmarks on the IBM POWER7 machine with synchronous_commit = >> on.  But, it didn't come out much better than 9.1.

Re: [HACKERS] new group commit behavior not helping?

2012-03-31 Thread Robert Haas
On Sat, Mar 31, 2012 at 8:31 PM, Peter Geoghegan wrote: > The exact benchmark that I ran was the update.sql pgbench-tools > benchmark, on my laptop. The idea was to produce a sympathetic > benchmark with a workload that was maximally commit-bound. Heikki > reproduced similar numbers on his laptop,

[HACKERS] new group commit behavior not helping?

2012-03-31 Thread Jeff Janes
On Saturday, March 31, 2012, Robert Haas wrote: > Hoping to demonstrate the wonders of our new group commit code, I ran > some benchmarks on the IBM POWER7 machine with synchronous_commit = > on. But, it didn't come out much better than 9.1. Where I would expect (and have seen) much improvement

Re: [HACKERS] measuring lwlock-related latency spikes

2012-03-31 Thread Robert Haas
On Sat, Mar 31, 2012 at 9:29 PM, Robert Haas wrote: > I've discovered a bug in my code that was causing it to print at most > 2 histogram buckets per lwlock, which obviously means that my previous > results were totally inaccurate.  Ah, the joys of benchmarking.  I > found the problem when I added

Re: [HACKERS] Command Triggers patch v18

2012-03-31 Thread Robert Haas
On Fri, Mar 30, 2012 at 11:19 AM, Dimitri Fontaine wrote: > Yeah context is not explicit, we could call that "toplevel": the command > tag of the command that the user typed. When toplevel is null, the event > trigger is fired on a command the user sent, when it's not null, the > trigger is fired

Re: [HACKERS] measuring lwlock-related latency spikes

2012-03-31 Thread Robert Haas
On Sat, Mar 31, 2012 at 6:01 PM, Greg Stark wrote: > On Sat, Mar 31, 2012 at 10:14 PM, Robert Haas wrote: >>> Isn't that lock held while doing visibility checks? >> >> Nope.  heap_update() and friends do a very complicated little dance to >> avoid that. > ... >>> What about I/O >>> waiting for a

Re: [HACKERS] new group commit behavior not helping?

2012-03-31 Thread Robert Haas
On Sat, Mar 31, 2012 at 8:31 PM, Peter Geoghegan wrote: > Why the low value for wal_writer_delay? A while back I was getting a benefit from cranking that down. I could try leaving it out and see if it matters. >> master: >> 01 tps = 118.968446 (including connections establishing) >> 02 tps = 12

Re: [HACKERS] new group commit behavior not helping?

2012-03-31 Thread Peter Geoghegan
On 1 April 2012 01:10, Robert Haas wrote: > Hoping to demonstrate the wonders of our new group commit code, I ran > some benchmarks on the IBM POWER7 machine with synchronous_commit = > on.  But, it didn't come out much better than 9.1.  pgbench, scale > factor 300, median of 3 30-minute test runs

[HACKERS] new group commit behavior not helping?

2012-03-31 Thread Robert Haas
Hoping to demonstrate the wonders of our new group commit code, I ran some benchmarks on the IBM POWER7 machine with synchronous_commit = on. But, it didn't come out much better than 9.1. pgbench, scale factor 300, median of 3 30-minute test runs, # clients = #threads, shared_buffers = 8GB, maint

Re: [HACKERS] measuring lwlock-related latency spikes

2012-03-31 Thread Greg Stark
On Sat, Mar 31, 2012 at 10:14 PM, Robert Haas wrote: >> Isn't that lock held while doing visibility checks? > > Nope.  heap_update() and friends do a very complicated little dance to > avoid that. ... >> What about I/O >> waiting for a clog page to be read? > > I'm pretty sure that can happen I'm

Re: [HACKERS] Re: [GENERAL] pg_dump incredibly slow dumping a single schema from a large db

2012-03-31 Thread Tom Lane
Mike Roest writes: > Any idea when 9.1.4 with this change will be out so we can pull the cluster > up. Well, we just did some releases last month, so unless somebody finds a really nasty security or data-loss issue, I'd think it will be a couple of months before the next set.

Re: [HACKERS] Publish checkpoint timing and sync files summary data to pg_stat_bgwriter

2012-03-31 Thread Peter Geoghegan
On 28 March 2012 15:23, Robert Haas wrote: > At any rate, I strongly agree that counting the number of > strategy allocations is not really a viable proxy for counting the > number of backend writes.  You can't know how many of those actually > got dirtied. Sure. > Since any backend write is nec

Re: [HACKERS] measuring lwlock-related latency spikes

2012-03-31 Thread Robert Haas
On Sat, Mar 31, 2012 at 8:58 AM, Greg Stark wrote: > On Sat, Mar 31, 2012 at 4:41 AM, Robert Haas wrote: >> But I didn't think we were ever supposed to hold content >> locks for that long. > > Isn't that lock held while doing visibility checks? Nope. heap_update() and friends do a very complica

[HACKERS] Re: [GENERAL] pg_dump incredibly slow dumping a single schema from a large db

2012-03-31 Thread Mike Roest
> > > I'm just pulling another backup using the stock 9.1.1 pg_dump to ensure > the backups are equivalent. > > Schema & data are identical between the 2 backups. the new backup passes all our tests for validating a tenant. Thank you again for the quick response! --Mike

[HACKERS] Re: [GENERAL] pg_dump incredibly slow dumping a single schema from a large db

2012-03-31 Thread Mike Roest
> > I've committed fixes for both these issues. If you are in a position to > test with 9.1 branch tip from git, it'd be nice to have confirmation > that these patches actually cure your problem. For both of them, the > issue seems to only show up in a subset of cases, which may explain why > we'

Re: [HACKERS] [GENERAL] pg_dump incredibly slow dumping a single schema from a large db

2012-03-31 Thread Tom Lane
I wrote: > So this is dumb; we should manage the "is the object already processed" > component of that with an O(1) check, like a bool array or some such, > rather than an O(N) search loop. > As for the getTables slowdown, the only part of that I can see that > looks to be both significant and ent

Re: [HACKERS] [COMMITTERS] pgsql: Add PGDLLIMPORT to ScanKeywords and NumScanKeywords.

2012-03-31 Thread Tom Lane
Andrew Dunstan writes: > On 03/31/2012 11:59 AM, Tom Lane wrote: >> The only simple fix I can see is to rename the symbols in ecpg and >> pg_dump to something else. This is probably a good thing anyway to >> reduce confusion. Anybody have another idea? > Seems the sane thing to do. Done, we'll

Re: [HACKERS] [COMMITTERS] pgsql: Add PGDLLIMPORT to ScanKeywords and NumScanKeywords.

2012-03-31 Thread Andrew Dunstan
On 03/31/2012 11:59 AM, Tom Lane wrote: Andrew Dunstan writes: On 03/31/2012 10:56 AM, Tom Lane wrote: Add PGDLLIMPORT to ScanKeywords and NumScanKeywords. Per buildfarm, this is now needed by contrib/pg_stat_statements. It seems to have broken mingw earlier now :-( Ugh. It looks like ec

Re: [HACKERS] [GENERAL] pg_dump incredibly slow dumping a single schema from a large db

2012-03-31 Thread Tom Lane
Mike Roest writes: > The file is 6 megs so I've dropped it here. > That was doing perf for the length of the pg_dump command and then a perf > report -n > http://dl.dropbox.com/u/13153/output.txt Hmm ... that's a remarkably verbose output format, but the useful part of this info seems to be just

Re: [HACKERS] [COMMITTERS] pgsql: Add PGDLLIMPORT to ScanKeywords and NumScanKeywords.

2012-03-31 Thread Tom Lane
Andrew Dunstan writes: > On 03/31/2012 10:56 AM, Tom Lane wrote: >> Add PGDLLIMPORT to ScanKeywords and NumScanKeywords. >> >> Per buildfarm, this is now needed by contrib/pg_stat_statements. > It seems to have broken mingw earlier now :-( Ugh. It looks like ecpg (and also pg_dump) are definin

Re: [HACKERS] Speed dblink using alternate libpq tuple storage

2012-03-31 Thread Tom Lane
Marko Kreen writes: > On Thu, Mar 29, 2012 at 06:56:30PM -0400, Tom Lane wrote: >> Yeah. Perhaps we should tweak the row-processor callback API so that >> it gets an explicit notification that "this is a new resultset". >> Duplicating PQexec's behavior would then involve having the dblink row >>

Re: [HACKERS] narwhal versus gnu_printf

2012-03-31 Thread Andrew Dunstan
On 03/31/2012 11:01 AM, Tom Lane wrote: I noticed that the build logs for buildfarm member narwhal (a mingw critter) contain an awful lot of occurrences of ../../src/include/utils/elog.h:159: warning: `gnu_printf' is an unrecognized format function type Evidently, the following hunk in pg_co

Re: [HACKERS] Tab completion of double quoted identifiers broken

2012-03-31 Thread Tom Lane
Dean Rasheed writes: > I just spotted that tab completion of double quoted identifiers seems > to be broken in 9.2devel. Yeah, looks like I broke it :-( --- I think I missed the fact that the last check with WORD_BREAKS was looking at the previous character not the current one, so I thought addin

[HACKERS] narwhal versus gnu_printf

2012-03-31 Thread Tom Lane
I noticed that the build logs for buildfarm member narwhal (a mingw critter) contain an awful lot of occurrences of ../../src/include/utils/elog.h:159: warning: `gnu_printf' is an unrecognized format function type Evidently, the following hunk in pg_config_manual.h failed to consider mingw. Is

Re: [HACKERS] measuring lwlock-related latency spikes

2012-03-31 Thread Simon Riggs
On Sat, Mar 31, 2012 at 1:58 PM, Greg Stark wrote: > On Sat, Mar 31, 2012 at 4:41 AM, Robert Haas wrote: >> But I didn't think we were ever supposed to hold content >> locks for that long. > > Isn't that lock held while doing visibility checks? What about I/O > waiting for a clog page to be read?

Re: [HACKERS] HTTP Frontend? (and a brief thought on materialized views)

2012-03-31 Thread Dobes Vandermeer
On Sat, Mar 31, 2012 at 1:44 AM, Daniel Farina wrote: > On Fri, Mar 30, 2012 at 10:21 AM, Daniel Farina wrote: > > Any enhancement here that can't be used with libpq via, say, drop-in > > .so seems unworkable to me, and that's why any solution that is > > basically proxying to the database is ba

Re: [HACKERS] Http Frontend implemented using pgsql?

2012-03-31 Thread Aidan Van Dyk
On Sat, Mar 31, 2012 at 6:27 AM, Dobes Vandermeer wrote: > I had a thought that it might be interesting to have a simple C fronted that > converts HTTP to and from some pgsql friendly structure and delegates all > the core logic to a stored procedure in the database. > > This might make it easier

Re: [HACKERS] measuring lwlock-related latency spikes

2012-03-31 Thread Greg Stark
On Sat, Mar 31, 2012 at 4:41 AM, Robert Haas wrote: > But I didn't think we were ever supposed to hold content > locks for that long. Isn't that lock held while doing visibility checks? What about I/O waiting for a clog page to be read? -- greg -- Sent via pgsql-hackers mailing list (pgsql-ha

Re: [HACKERS] Patch pg_is_in_backup()

2012-03-31 Thread Gabriele Bartolini
Hi Gilles, first and foremost, sorry for jumping in this thread so late. I read all previous discussions and I'd be happy to help you with this patch. Agreed and sorry for the response delay. I've attached 2 patches here, the first one is the same as before with just the renaming of the f

Re: [HACKERS] Odd out of memory problem.

2012-03-31 Thread Peter Eisentraut
On fre, 2012-03-30 at 22:59 -0700, Hitoshi Harada wrote: > I don't know exactly how PL/Proxy or pgpool accomplish the multi-phase > aggregate, They don't. > but in theory the proposal above is state-merge function, > so it doesn't apply to general aggregate results that passed through > the final

Re: [HACKERS] measuring lwlock-related latency spikes

2012-03-31 Thread Robert Haas
On Sat, Mar 31, 2012 at 4:53 AM, Simon Riggs wrote: > The next logical step in measuring lock waits is to track the reason > for the lock wait, not just the lock wait itself. I had the same thought. I'm not immediately sure what the best way to do that is, but I'll see if I can figure something

[HACKERS] Http Frontend implemented using pgsql?

2012-03-31 Thread Dobes Vandermeer
I had a thought that it might be interesting to have a simple C fronted that converts HTTP to and from some pgsql friendly structure and delegates all the core logic to a stored procedure in the database. This might make it easier to hack on the API without worrying about memory management and buf

Re: [HACKERS] measuring lwlock-related latency spikes

2012-03-31 Thread Simon Riggs
On Sat, Mar 31, 2012 at 4:41 AM, Robert Haas wrote: > which means, if I'm not > confused here, that every single lwlock-related stall > 1s happened > while waiting for a buffer content lock.  Moreover, each event > affected a different buffer.  I find this result so surprising that I > have a har

[HACKERS] Tab completion of double quoted identifiers broken

2012-03-31 Thread Dean Rasheed
Hi, I just spotted that tab completion of double quoted identifiers seems to be broken in 9.2devel. For example things like this which worked in 9.1 no longer work: UPDATE "foo bar" It looks like the problem is in get_previous_words() here: if (buf[start] == '"')