Re: [HACKERS] Synch Rep: communication between backends and walsender

2009-06-22 Thread Fujii Masao
Hi, On Tue, Jun 23, 2009 at 1:55 AM, Markus Wanner wrote: >> On the other hand, I think that a semaphore should be used instead of a >> signal >> at least for backends to wait for walsender. This would turn around >> the situation >> on HPUX in some degree. In this case, the remaining problem is

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Simon Riggs
On Mon, 2009-06-22 at 15:18 -0500, Kevin Grittner wrote: > Tom Lane wrote: > > > default postgresql.conf (comments stripped) > max_connections = 100 > shared_buffers = 32MB This forces ring size to be 4MB, since min(32MB/8, ringsize). Please re-run tests with your config, ring size 4MB, wh

[HACKERS] PK not being restored

2009-06-22 Thread Rajdeep Das
Dear Sir/Madam, I am using postgresql db server for our production environment. I have created a schema by the name “application”. The public schema is not being used at all. I have noticed that suddenly, the public schema is having some table definition and a whole lot of functions. I have no i

Re: [HACKERS] security checks for largeobjects?

2009-06-22 Thread KaiGai Kohei
David Fetter wrote: > On Tue, Jun 23, 2009 at 10:38:59AM +0900, KaiGai Kohei wrote: >> David Fetter wrote: >>> On Mon, Jun 22, 2009 at 11:31:45AM -0400, Tom Lane wrote: David Fetter writes: > On Mon, Jun 22, 2009 at 05:18:51PM +0300, Peter Eisentraut wrote: >> MED is management of ext

Re: [HACKERS] building without perl

2009-06-22 Thread Andrew Dunstan
Tom Lane wrote: Andrew Dunstan writes: Tom Lane wrote: Actually, further investigation discloses that someone broke the no-Perl defense in psql's makefile back in 8.0, and we hadn't noticed till now. Don't we ship sql_help.h in the tarball? I thought that's all the perl

Re: [HACKERS] security checks for largeobjects?

2009-06-22 Thread KaiGai Kohei
KaiGai Kohei wrote: > KaiGai Kohei wrote: >> The todo list says, as follows: >> * Binary Data >> o Add security checks for large objects >> >> http://wiki.postgresql.org/wiki/Todo#Binary_Data >> >> Is anyone working on? or interested in? > > OK, it seems to me nobody is working on the todo item.

Re: [HACKERS] security checks for largeobjects?

2009-06-22 Thread David Fetter
On Tue, Jun 23, 2009 at 10:38:59AM +0900, KaiGai Kohei wrote: > David Fetter wrote: > > On Mon, Jun 22, 2009 at 11:31:45AM -0400, Tom Lane wrote: > >> David Fetter writes: > >>> On Mon, Jun 22, 2009 at 05:18:51PM +0300, Peter Eisentraut wrote: > MED is management of external data, whereas the

Re: [HACKERS] building without perl

2009-06-22 Thread Tom Lane
Andrew Dunstan writes: > Tom Lane wrote: >> Actually, further investigation discloses that someone broke the no-Perl >> defense in psql's makefile back in 8.0, and we hadn't noticed till now. > Don't we ship sql_help.h in the tarball? I thought that's all the perl > was used for in psql. Yeah,

Re: [HACKERS] building without perl

2009-06-22 Thread Andrew Dunstan
Tom Lane wrote: Actually, further investigation discloses that someone broke the no-Perl defense in psql's makefile back in 8.0, and we hadn't noticed till now. So it certainly seems to be the case that nobody associated with the Postgres project works in a Perl-free environment. Nonetheless,

Re: [HACKERS] building without perl

2009-06-22 Thread Tom Lane
Robert Haas writes: > So, is our policy that it's OK to use perl to generate things, as long > as they are things that are architecture-independent and > configuration-independent therefore could be pregenerated for the > release tarball? If so, it seems we might be able to punt > Gen_fmgrtab.sh

Re: [HACKERS] building without perl

2009-06-22 Thread Robert Haas
On Mon, Jun 22, 2009 at 10:12 PM, Tom Lane wrote: > Robert Haas writes: >> I'm a bit confused as to what we're trying to support. > > You are not supposed to need perl to build from a release tarball > (except on Windows which doesn't have a decent make or shell, so we > depend on perl instead).  

Re: [HACKERS] building without perl

2009-06-22 Thread Tom Lane
Robert Haas writes: > I'm a bit confused as to what we're trying to support. You are not supposed to need perl to build from a release tarball (except on Windows which doesn't have a decent make or shell, so we depend on perl instead). You do need it to build from a CVS checkout. Same story as b

[HACKERS] building without perl

2009-06-22 Thread Robert Haas
I'm a bit confused as to what we're trying to support. While I was poking around tonight, I notice that we have both ./src/backend/utils/Gen_fmgrtab.sh and ./src/backend/utils/Gen_fmgrtab.pl, ostensibly because: # The reason for implementing this functionality twice is that we don't # require peo

Re: [HACKERS] security checks for largeobjects?

2009-06-22 Thread KaiGai Kohei
David Fetter wrote: > On Mon, Jun 22, 2009 at 11:31:45AM -0400, Tom Lane wrote: >> David Fetter writes: >>> On Mon, Jun 22, 2009 at 05:18:51PM +0300, Peter Eisentraut wrote: MED is management of external data, whereas the large objects are internal, no? >>> It depends on your definition.

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Alan Li
On Mon, Jun 22, 2009 at 7:16 AM, Tom Lane wrote: > Alan Li writes: > > How much concern is there for the contention for use cases where the WAL > > can't be bypassed? > > If you mean "is something going to be done about it in 8.4", the > answer is "no". This is a pre-existing issue that there i

Re: [HACKERS] Changed error message for blocks by prepared transactions

2009-06-22 Thread Tom Lane
"Andreas 'ads' Scherbaum" writes: > the small attached patch changes the error message for a blocked > database in case there are prepared transactions. Isn't this duplicative of the errdetail_busy_db code? And anyway I do not see a reason not to consider prepared transactions as "other users".

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Tom Lane
"Kevin Grittner" writes: > The checkpoint_segments seems dramatic enough to be real. I wonder if > the test is short enough that it never got around to re-using any of > them, so it was doing extra writes for the initial creation during the > test? That's exactly what I was about to suggest. Ar

[HACKERS] Changed error message for blocks by prepared transactions

2009-06-22 Thread Andreas 'ads' Scherbaum
Hello, the small attached patch changes the error message for a blocked database in case there are prepared transactions. The original message "accessed by other users" is misleading. Example: - snip - postgres=# begin; BEGIN postgres=# prepare transaction 'abc'; PREPARE TRANSACTION po

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Kevin Grittner
I wrote: > Stefan Kaltenbrunner wrote: > >> A 25-30% performance regression in our main bulk loading mechanism >> should at least be explained before the release... > > I think a performance regression of that magnitude merits holding > up a release to resolve. Wow. That sure took a long

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Kevin Grittner
Tom Lane wrote: > Huh, that's bizarre. I can see that increasing shared_buffers > should make no difference in this test case (we're not using them > all anyway). But why should increasing wal_buffers make it slower? > I forget the walwriter's control algorithm at the moment ... maybe > it wo

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Kevin Grittner
Stefan Kaltenbrunner wrote: > A 25-30% performance regression in our main bulk loading mechanism > should at least be explained before the release... I think a performance regression of that magnitude merits holding up a release to resolve. Note that in a follow-up post showing the slow 8.

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Kevin Grittner
Greg Smith wrote: > Do you have happen to have a build with assertions turned on? Nope. I showed my ./configure options upthread, but can confirm with pg_config: BINDIR = /usr/local/pgsql-8.4rc1/bin DOCDIR = /usr/local/pgsql-8.4rc1/share/doc HTMLDIR = /usr/local/pgsql-8.4rc1/share/doc INCL

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Greg Smith
On Mon, 22 Jun 2009, Kevin Grittner wrote: When I tried with setting the ring size to 16MB, I accidentally left off the step to copy the postgresql.conf file, and got better performance. Do you have happen to have a build with assertions turned on? That is one common cause of performance goi

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Tom Lane
"Kevin Grittner" writes: > Tom Lane wrote: >> I wonder though whether the wal_buffers setting interacts with the >> ring size. Has everyone who's tested this used the same 16MB >> wal_buffers setting as in Alan's original scenario? > I had been using his postgresql.conf file, then added autov

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Kevin Grittner
Tom Lane wrote: > I wonder though whether the wal_buffers setting interacts with the > ring size. Has everyone who's tested this used the same 16MB > wal_buffers setting as in Alan's original scenario? I had been using his postgresql.conf file, then added autovacuum = off. When I tried with

Re: [HACKERS] [BUGS] BUG #4862: different results in to_date() between 8.3.7 & 8.4.RC1

2009-06-22 Thread Tom Lane
Brendan Jurd writes: > Here's a one-line patch to fix a regression in the new from_char code > I introduced into 8.4. > Versions <= 8.3 skipped over any whitespace immediately preceding any > integer field, and this behaviour was lost in HEAD when my from_char > patch was committed back in Septem

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Dimitri Fontaine
Le 22 juin 2009 à 17:24, Tom Lane a écrit : Simon Riggs writes: I was thinking it might be beneficial to be able to defer writing WAL until COPY is complete, so heap_sync would either fsync the whole heap file or copy the whole file to WAL. What about indexes? Skip this optimisation if

[HACKERS] Re: [BUGS] BUG #4862: different results in to_date() between 8.3.7 & 8.4.RC1

2009-06-22 Thread Brendan Jurd
2009/6/23 Tom Lane : > Brendan Jurd writes: >> I should be able to get the same results by snipping an extra >> strspace_len() characters in the new code path in >> from_char_parse_int_len().  This ought to be a one-line fix that >> doesn't clobber the good parts of my work so far.  I'll run some

Re: [HACKERS] security checks for largeobjects?

2009-06-22 Thread David Fetter
On Mon, Jun 22, 2009 at 11:31:45AM -0400, Tom Lane wrote: > David Fetter writes: > > On Mon, Jun 22, 2009 at 05:18:51PM +0300, Peter Eisentraut wrote: > >> MED is management of external data, whereas the large objects are > >> internal, no? > > > It depends on your definition. The lo interface i

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Tom Lane
Heikki Linnakangas writes: > Tom Lane wrote: >> I'm not convinced that WAL segment boundaries are particularly relevant >> to this. The unit of flushing is an 8K page, not a segment. > We fsync() the old WAL segment every time we switch to a new WAL > segment. That's what I meant by "flush". >

Re: [HACKERS] Synch Rep: communication between backends and walsender

2009-06-22 Thread Markus Wanner
Hi, Fujii Masao wrote: > The perfect solution seems to be to remove SA_RESTART and handle EINTR > in an appropriate way after every syscalls. But, this is very tough job and > has > much influence on whole source code, so I don't think that this should be done > as part of synch rep. Especially

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Andrew Dunstan
Tom Lane wrote: Andrew Dunstan writes: Tom Lane wrote: I thought he was asking for a solution to the problem of WALInsertLock contention. In any case, we have "WAL bypass on a table by table basis" now, don't we? If we do I'm ignorant of it ;-) How do we say "Never WAL

Re: [HACKERS] security checks for largeobjects?

2009-06-22 Thread Tom Lane
David Fetter writes: > On Mon, Jun 22, 2009 at 05:18:51PM +0300, Peter Eisentraut wrote: >> MED is management of external data, whereas the large objects are internal, >> no? > It depends on your definition. The lo interface is pretty much to > objects on the file system directly. LO's are tra

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Simon Riggs
On Mon, 2009-06-22 at 11:24 -0400, Tom Lane wrote: > Simon Riggs writes: > > I was thinking it might be beneficial to be able to defer writing WAL > > until COPY is complete, so heap_sync would either fsync the whole heap > > file or copy the whole file to WAL. > > What about indexes? I was thi

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Tom Lane
Simon Riggs writes: > I was thinking it might be beneficial to be able to defer writing WAL > until COPY is complete, so heap_sync would either fsync the whole heap > file or copy the whole file to WAL. What about indexes? regards, tom lane -- Sent via pgsql-hackers mai

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Simon Riggs
On Mon, 2009-06-22 at 11:14 -0400, Tom Lane wrote: > Stephen Frost writes: > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > >> The more useful case for data load is "create or truncate it in the > >> same transaction", of course. > > > Unfortunately, WAL bypass also requires not being in archive mod

Re: [HACKERS] security checks for largeobjects?

2009-06-22 Thread David Fetter
On Mon, Jun 22, 2009 at 05:18:51PM +0300, Peter Eisentraut wrote: > On Monday 22 June 2009 16:24:44 David Fetter wrote: > > On Mon, Jun 22, 2009 at 02:00:10PM +0900, KaiGai Kohei wrote: > > > KaiGai Kohei wrote: > > > > The todo list says, as follows: > > > > * Binary Data > > > > o Add security

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > Unfortunately, WAL bypass also requires not being in archive mode with > > no way to turn that off w/o a server restart, aiui. > > Well, if you're trying to archive then you certainly wouldn't want WAL > off, so I'm failing to see

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Tom Lane
Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> The more useful case for data load is "create or truncate it in the >> same transaction", of course. > Unfortunately, WAL bypass also requires not being in archive mode with > no way to turn that off w/o a server restart, aiui. We

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: > The more useful case for data load is "create or truncate it in the > same transaction", of course. Unfortunately, WAL bypass also requires not being in archive mode with no way to turn that off w/o a server restart, aiui. Thanks,

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Aidan Van Dyk
* Andrew Dunstan [090622 10:47]: > If we do I'm ignorant of it ;-) How do we say "Never WAL this table"? CREATE TEMPORARY TABLE ... a. -- Aidan Van Dyk Create like a god, ai...@highrise.ca command like a king,

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Tom Lane
Andrew Dunstan writes: > Tom Lane wrote: >> I thought he was asking for a solution to the problem of WALInsertLock >> contention. In any case, we have "WAL bypass on a table by table basis" >> now, don't we? > If we do I'm ignorant of it ;-) How do we say "Never WAL this table"? Make it a tempo

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Andrew Dunstan
Tom Lane wrote: I thought he was asking for a solution to the problem of WALInsertLock contention. In any case, we have "WAL bypass on a table by table basis" now, don't we? If we do I'm ignorant of it ;-) How do we say "Never WAL this table"? cheers andrew --

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Tom Lane
Andrew Dunstan writes: > Tom Lane wrote: >> Alan Li writes: >>> How much concern is there for the contention for use cases where the WAL >>> can't be bypassed? >> >> If you mean "is something going to be done about it in 8.4", the >> answer is "no". This is a pre-existing issue that there is no

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Andrew Dunstan
Tom Lane wrote: Alan Li writes: How much concern is there for the contention for use cases where the WAL can't be bypassed? If you mean "is something going to be done about it in 8.4", the answer is "no". This is a pre-existing issue that there is no simple fix for.

Re: [HACKERS] security checks for largeobjects?

2009-06-22 Thread Peter Eisentraut
On Monday 22 June 2009 16:24:44 David Fetter wrote: > On Mon, Jun 22, 2009 at 02:00:10PM +0900, KaiGai Kohei wrote: > > KaiGai Kohei wrote: > > > The todo list says, as follows: > > > * Binary Data > > > o Add security checks for large objects > > > > > > http://wiki.postgresql.org/wiki/Todo#Bina

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Tom Lane
Alan Li writes: > How much concern is there for the contention for use cases where the WAL > can't be bypassed? If you mean "is something going to be done about it in 8.4", the answer is "no". This is a pre-existing issue that there is no simple fix for. regards, tom lan

Re: [HACKERS] security checks for largeobjects?

2009-06-22 Thread David Fetter
On Mon, Jun 22, 2009 at 02:00:10PM +0900, KaiGai Kohei wrote: > KaiGai Kohei wrote: > > The todo list says, as follows: > > * Binary Data > > o Add security checks for large objects > > > > http://wiki.postgresql.org/wiki/Todo#Binary_Data > > > > Is anyone working on? or interested in? > > OK,

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Simon Riggs
On Mon, 2009-06-22 at 10:52 +0300, Heikki Linnakangas wrote: > Tom Lane wrote: > > Heikki Linnakangas writes: > >> I was going to say that since we flush the WAL every 16MB anyway (at > >> every XLOG file switch), you shouldn't see any benefit with larger ring > >> buffers, since to fill 16MB o

Re: [HACKERS] 8.4 open item: copy performance regression?

2009-06-22 Thread Heikki Linnakangas
Tom Lane wrote: Heikki Linnakangas writes: I was going to say that since we flush the WAL every 16MB anyway (at every XLOG file switch), you shouldn't see any benefit with larger ring buffers, since to fill 16MB of data you're not going to write more than 16MB WAL. I'm not convinced that WA

Re: [HACKERS] Synch Rep: communication between backends and walsender

2009-06-22 Thread Fujii Masao
Hi, On Sat, Jun 20, 2009 at 6:05 PM, Markus Wanner wrote: > Reading the past messages on this topic, I realized that this problem so > far only affects HPUX. I fear the proposed UDP/semaphores approach might > have a similar gotcha on at least one of the supported platforms, too. > Limits of open

Re: [HACKERS] Missing Docs for MOVE direction?

2009-06-22 Thread Peter Eisentraut
On Monday 22 June 2009 03:15:57 David E. Wheeler wrote: > On Jun 21, 2009, at 5:07 PM, David E. Wheeler wrote: > > I was just looking at the documentation for cursors command, and > > noticed that the MOVE command's "direction" argument doesn't seem to > > have documentation for its possible values