Re: [HACKERS] Group Commit

2007-04-10 Thread Zeugswetter Andreas ADI SD
> > > I've been working on the patch to enhance our group commit behavior. > > > The patch is a dirty hack at the moment, but I'm settled on the > > > algorithm I'm going to use and I know the issues involved. > > > > One question that just came to mind is whether Simon's no-commit-wait > > pa

Re: [HACKERS] Group Commit

2007-04-10 Thread Heikki Linnakangas
Tom Lane wrote: Heikki Linnakangas <[EMAIL PROTECTED]> writes: I've been working on the patch to enhance our group commit behavior. The patch is a dirty hack at the moment, but I'm settled on the algorithm I'm going to use and I know the issues involved. One question that just came to mind is

Re: [HACKERS] Anyone interested in improving postgresql scaling?

2007-04-10 Thread Mark Kirkwood
Kris Kennaway wrote: If so, then your task is the following: Make SYSV semaphores less dumb about process wakeups. Currently whenever the semaphore state changes, all processes sleeping on the semaphore are woken, even if we only have released enough resources for one waiting process to claim.

[HACKERS] \da doesn't show result type

2007-04-10 Thread Peter Eisentraut
Is there a reason that \da doesn't show the functions result type, like \df does? I would find that information useful. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 1: if posting/reading through Usenet

Re: [HACKERS] "select ('{}'::text[])[1]" returns NULL -- is it correct?

2007-04-10 Thread Tom Lane
"Nikolay Samokhvalov" <[EMAIL PROTECTED]> writes: > I remember several cases when people (e.g. me :-) ) were spending some > time trying to find an error in some pl/pgsql function and the reason > lied in incorrect work with arrays (i.e. messages like "index is out > of bounds" and "index cannot be

[HACKERS] Idle idea for a feature

2007-04-10 Thread Tom Lane
psql's \d command tells you about outgoing foreign key constraints (ie, ones referencing another table from this one). It doesn't tell you about incoming ones (ie, ones where another table references this one). ISTM it'd be a good idea if it did, as "are there any incoming foreign keys" seems to

Re: [HACKERS] Anyone interested in improving postgresql scaling?

2007-04-10 Thread Tom Lane
Mark Kirkwood <[EMAIL PROTECTED]> writes: > Kris Kennaway wrote: >> If so, then your task is the following: >> >> Make SYSV semaphores less dumb about process wakeups. Currently >> whenever the semaphore state changes, all processes sleeping on the >> semaphore are woken, even if we only have rel

Re: [HACKERS] TOASTing smaller things

2007-04-10 Thread Luke Lonergan
Hi Bruce, How about these: - Allow specification of TOAST size threshold in bytes on a per column basis - Enable storage of columns in separate TOAST tables - Enable use of multi-row compression method(s) for TOAST tables - Luke On 3/26/07 5:39 PM, "Bruce Momjian" <[EMAIL PROTECTED]> wrote: >

Re: [HACKERS] \da doesn't show result type

2007-04-10 Thread Tom Lane
Peter Eisentraut <[EMAIL PROTECTED]> writes: > Is there a reason that \da doesn't show the functions result type, like \df > does? I would find that information useful. +1, I've been annoyed by that too. regards, tom lane ---(end of broadcast)---

Re: [HACKERS] Idle idea for a feature

2007-04-10 Thread Stephen Frost
* Tom Lane ([EMAIL PROTECTED]) wrote: > one). ISTM it'd be a good idea if it did, as "are there any incoming > foreign keys" seems to be a question we constantly ask when solving > update-performance problems, and there isn't any easy way to check for > such. I'm not real sure what the printout s

Re: [HACKERS] \da doesn't show result type

2007-04-10 Thread Magnus Hagander
On Tue, Apr 10, 2007 at 10:32:38AM -0400, Tom Lane wrote: > Peter Eisentraut <[EMAIL PROTECTED]> writes: > > Is there a reason that \da doesn't show the functions result type, like \df > > does? I would find that information useful. > > +1, I've been annoyed by that too. Eh, but it does? As of

Re: [HACKERS] \da doesn't show result type

2007-04-10 Thread Peter Eisentraut
Am Dienstag, 10. April 2007 17:12 schrieb Magnus Hagander: > > > Is there a reason that \da doesn't show the functions result type, like > > > \df does? I would find that information useful. > Eh, but it does? As of > http://archives.postgresql.org/pgsql-committers/2007-03/msg00138.php. Ah, OK,

Re: [HACKERS] Group Commit

2007-04-10 Thread Tom Lane
Heikki Linnakangas <[EMAIL PROTECTED]> writes: > I've refrained from spending time on group commit until the > commit-no-wait patch lands, because it's going to conflict anyway. I'm > starting to feel we should not try to rush group commit into 8.3, unless > it somehow falls out of the commit-no

Re: [HACKERS] [DOCS] uuid type not documented

2007-04-10 Thread Peter Eisentraut
Am Dienstag, 10. April 2007 17:30 schrieb Neil Conway: > On Tue, 2007-04-10 at 17:24 +0200, Peter Eisentraut wrote: > > The new uuid type is lacking documentation. > > We had also talked about including some UUID generation functionality in > 8.3, but it should be okay to leave that for 8.4. The p

Re: [HACKERS] [DOCS] uuid type not documented

2007-04-10 Thread Joshua D. Drake
Peter Eisentraut wrote: Am Dienstag, 10. April 2007 17:30 schrieb Neil Conway: On Tue, 2007-04-10 at 17:24 +0200, Peter Eisentraut wrote: The new uuid type is lacking documentation. We had also talked about including some UUID generation functionality in 8.3, but it should be okay to leave tha

Re: [HACKERS] [DOCS] uuid type not documented

2007-04-10 Thread Tom Lane
"Joshua D. Drake" <[EMAIL PROTECTED]> writes: > Peter Eisentraut wrote: >> The problem is that most of the standard methods are platform dependent, as >> they require MAC addresses or a "good" random source, for instance. I'm not >> sure how we wanted to solve that, but certainly leaving the uuid

Re: [PATCHES] [HACKERS] Full page writes improvement, code update

2007-04-10 Thread Tom Lane
Koichi Suzuki <[EMAIL PROTECTED]> writes: > My proposal is to remove unnecessary full page writes (they are needed > in crash recovery from inconsistent or partial writes) when we copy WAL > to archive log and rebuilt them as a dummy when we restore from archive > log. > ... > Benchmark: DBT-2 >

Re: [HACKERS] [PATCHES] Fix mdsync never-ending loop problem

2007-04-10 Thread Tom Lane
I wrote: > This patch looks fairly sane to me; I have a few small gripes about > coding style but that can be fixed while applying. Heikki, you were > concerned about the cycle-ID idea; do you have any objection to this > patch? Actually, on second look I think the key idea here is Takahiro-san's

Re: [HACKERS] Anyone interested in improving postgresql scaling?

2007-04-10 Thread Tom Lane
Kris Kennaway <[EMAIL PROTECTED]> writes: >>> Make SYSV semaphores less dumb about process wakeups. Currently >>> whenever the semaphore state changes, all processes sleeping on the >>> semaphore are woken, even if we only have released enough resources >>> for one waiting process to claim. >> Co

Re: [PATCHES] [HACKERS] [Fwd: Index Advisor]

2007-04-10 Thread Gurjeet Singh
Hi Tom, The original patch was submitted by Kai Sattler, and we (at EDB) spent a lot of time improving it, making it as seamless and as user-friendly as possible. As is evident from the version number of the patch (v26), it has gone through a lot of iterations, and was available to the communi

Re: [HACKERS] [PATCHES] Fix mdsync never-ending loop problem

2007-04-10 Thread ITAGAKI Takahiro
(Sorry if you receive duplicate messages. I resend it since it was not delivered after a day.) Here is another patch to fix never-ending loop in mdsync. I introduced a mdsync counter (cycle id) and cancel flags to fix the problem. The mdsync counter is incremented at the every beginning of mdsy

Re: [HACKERS] [PATCHES] Fix mdsync never-ending loop problem

2007-04-10 Thread Tom Lane
ITAGAKI Takahiro <[EMAIL PROTECTED]> writes: > Here is another patch to fix never-ending loop in mdsync. I introduced > a mdsync counter (cycle id) and cancel flags to fix the problem. > The mdsync counter is incremented at the every beginning of mdsync(). > Each pending entry has a field assigned

Re: [HACKERS] Anyone interested in improving postgresql scaling?

2007-04-10 Thread Kris Kennaway
On Tue, Apr 10, 2007 at 10:41:04PM +1200, Mark Kirkwood wrote: > Kris Kennaway wrote: > >If so, then your task is the following: > > > >Make SYSV semaphores less dumb about process wakeups. Currently > >whenever the semaphore state changes, all processes sleeping on the > >semaphore are woken, eve

Re: [HACKERS] Anyone interested in improving postgresql scaling?

2007-04-10 Thread Kris Kennaway
On Tue, Apr 10, 2007 at 10:23:42AM -0400, Tom Lane wrote: > Mark Kirkwood <[EMAIL PROTECTED]> writes: > > Kris Kennaway wrote: > >> If so, then your task is the following: > >> > >> Make SYSV semaphores less dumb about process wakeups. Currently > >> whenever the semaphore state changes, all proc

[HACKERS] prepared statements logging

2007-04-10 Thread marcofuics
Hi * I am using the postgresql-8.2.3, with a jdbc-8.2-504 (the GeoNet webServer tool...) My question is : Is the <> server able to log the whole <> query? {made by a prepared statement} Looking at the log I can see only queries of the type: (cut). LOG: execute : SELECT * FROM

Re: [HACKERS] "select ('{}'::text[])[1]" returns NULL -- is it correct?

2007-04-10 Thread Nikolay Samokhvalov
On 4/9/07, Tom Lane <[EMAIL PROTECTED]> wrote: "Nikolay Samokhvalov" <[EMAIL PROTECTED]> writes: > As I can see here, when I ask for element that doesn't exist, the > database returns NULL for me. Maybe it's well-known issue (and > actually I understood this behaviour before), but strictly speaki

Re: [HACKERS] "select ('{}'::text[])[1]" returns NULL -- is it correct?

2007-04-10 Thread Nikolay Samokhvalov
On 4/10/07, Tom Lane <[EMAIL PROTECTED]> wrote: "Nikolay Samokhvalov" <[EMAIL PROTECTED]> writes: > I remember several cases when people (e.g. me :-) ) were spending some > time trying to find an error in some pl/pgsql function and the reason > lied in incorrect work with arrays (i.e. messages li

Re: [HACKERS] [DOCS] uuid type not documented

2007-04-10 Thread Neil Conway
On Tue, 2007-04-10 at 18:28 +0200, Peter Eisentraut wrote: > The problem is that most of the standard methods are platform dependent, as > they require MAC addresses or a "good" random source, for instance. http://archives.postgresql.org/pgsql-patches/2007-01/msg00392.php ISTM random() or simila

Re: [HACKERS] [DOCS] uuid type not documented

2007-04-10 Thread Andrew Dunstan
Neil Conway wrote: On Tue, 2007-04-10 at 18:28 +0200, Peter Eisentraut wrote: The problem is that most of the standard methods are platform dependent, as they require MAC addresses or a "good" random source, for instance. http://archives.postgresql.org/pgsql-patches/2007-01/msg00392.ph

Re: [HACKERS] "select ('{}'::text[])[1]" returns NULL -- is it correct?

2007-04-10 Thread Florian G. Pflug
Nikolay Samokhvalov wrote: On 4/10/07, Tom Lane <[EMAIL PROTECTED]> wrote: "Nikolay Samokhvalov" <[EMAIL PROTECTED]> writes: > I remember several cases when people (e.g. me :-) ) were spending some > time trying to find an error in some pl/pgsql function and the reason > lied in incorrect work w

Re: [HACKERS] Anyone interested in improving postgresql scaling?

2007-04-10 Thread Tom Lane
Kris Kennaway <[EMAIL PROTECTED]> writes: > On Tue, Apr 10, 2007 at 02:46:56PM -0400, Tom Lane wrote: >> Oh, I'm sure the BSD kernel acts as you describe. But Mark's point is >> that Postgres never has more than one process waiting on any particular >> SysV semaphore, and so the problem doesn't re

Re: [HACKERS] [DOCS] uuid type not documented

2007-04-10 Thread Florian G. Pflug
Neil Conway wrote: On Tue, 2007-04-10 at 18:28 +0200, Peter Eisentraut wrote: The problem is that most of the standard methods are platform dependent, as they require MAC addresses or a "good" random source, for instance. http://archives.postgresql.org/pgsql-patches/2007-01/msg00392.php ISTM

Re: [HACKERS] Idle idea for a feature

2007-04-10 Thread Guillaume Smet
On 4/10/07, Tom Lane <[EMAIL PROTECTED]> wrote: ISTM it'd be a good idea if it did, as "are there any incoming foreign keys" seems to be a question we constantly ask when solving update-performance problems, and there isn't any easy way to check for such. Sure. We wrote a stored proc to do that

Re: [HACKERS] [DOCS] uuid type not documented

2007-04-10 Thread Tom Lane
Neil Conway <[EMAIL PROTECTED]> writes: > On Tue, 2007-04-10 at 18:28 +0200, Peter Eisentraut wrote: >> The problem is that most of the standard methods are platform dependent, as >> they require MAC addresses or a "good" random source, for instance. > http://archives.postgresql.org/pgsql-patches

Re: [HACKERS] Anyone interested in improving postgresql scaling?

2007-04-10 Thread Tom Lane
Kris Kennaway <[EMAIL PROTECTED]> writes: > I have not studied the exact code path, but there are indeed multiple > wakeups happening from the semaphore code (as many as the number of > active postgresql processes). It is easy to instrument > sleepq_broadcast() and log them when they happen. Ther

Re: [HACKERS] Anyone interested in improving postgresql scaling?

2007-04-10 Thread Andrew - Supernews
On 2007-04-10, Tom Lane <[EMAIL PROTECTED]> wrote: > Kris Kennaway <[EMAIL PROTECTED]> writes: >> I have not studied the exact code path, but there are indeed multiple >> wakeups happening from the semaphore code (as many as the number of >> active postgresql processes). It is easy to instrument >

Re: [HACKERS] Anyone interested in improving postgresql scaling?

2007-04-10 Thread Tom Lane
Kris Kennaway <[EMAIL PROTECTED]> writes: > On Tue, Apr 10, 2007 at 05:36:17PM -0400, Tom Lane wrote: >> Anyway I'd be interested to know what the test case is, and which PG >> version you were testing. > I used 8.2 (and some older version when I first noticed it a year ago) > and either sysbench

Re: [HACKERS] [EMAIL PROTECTED]: Re: Anyone interested in improving postgresql scaling?]

2007-04-10 Thread Tom Lane
Kris Kennaway <[EMAIL PROTECTED]> forwards: > Yes but there are still a lot of wakeups to be avoided in the current > System V semaphore code. More specifically, not only do we wakeup all > the processes waiting on a single semaphore everytime something changes, > but we also wakeup all processes

Re: [HACKERS] [EMAIL PROTECTED]: Re: Anyone interested in improving postgresql scaling?]

2007-04-10 Thread Tom Lane
Maxime Henrion <[EMAIL PROTECTED]> writes: > Thanks for forwarding my mail, Kris! To Tom: if you can get my mails > to reach pgsql-hackers@ somehow that would be just great :-). They'll get approved eventually, just like mine to the BSD lists will get approved eventually ;-) >> The only thing we

Re: [HACKERS] [EMAIL PROTECTED]: Re: Anyone interested in improving postgresql scaling?]

2007-04-10 Thread Mark Kirkwood
Tom Lane wrote: I think the high number of setproctitle() calls are more problematic to us at the moment, Kris can comment on that. As of PG 8.2 it is possible to turn those off. I don't think there's a lot of enthusiasm for turning them off by default ... at least not yet. But it might mak

Re: [PATCHES] [HACKERS] Full page writes improvement, code update

2007-04-10 Thread Koichi Suzuki
Hi, In the case below, we run DBT-2 benchmark for one hour to get the measure. Checkpoint occured three times (checkpoint interval was 20min). For more information, when checkpoint interval is one hour, the amount of the archived log size was as follows: cp: 3.1GB gzip: 1.

Re: [PATCHES] [HACKERS] Full page writes improvement, code update

2007-04-10 Thread Joshua D. Drake
> In terms of idle time for gzip and other command to archive WAL offline, > no difference in the environment was given other than the command to > archive. My guess is because the user time is very large in gzip, it > has more chance for scheduler to give resource to other processes. In > the

Re: [HACKERS] [EMAIL PROTECTED]: Re: Anyone interested in improving postgresql scaling?]

2007-04-10 Thread Tom Lane
Kris Kennaway <[EMAIL PROTECTED]> writes: > I think the high number of setproctitle() calls are more problematic > to us at the moment, Kris can comment on that. > Since we've basically had it handed to us that calling setproctitle() > thousands of times per second is something that real a

Re: [HACKERS] [DOCS] uuid type not documented

2007-04-10 Thread Roman Neuhauser
# [EMAIL PROTECTED] / 2007-04-10 15:49:08 -0400: > Neil Conway wrote: > >On Tue, 2007-04-10 at 18:28 +0200, Peter Eisentraut wrote: > > > >>The problem is that most of the standard methods are platform dependent, > >>as they require MAC addresses or a "good" random source, for instance. > >>

Re: [PATCHES] [HACKERS] Full page writes improvement, code update

2007-04-10 Thread Hannu Krosing
Ühel kenal päeval, T, 2007-04-10 kell 18:17, kirjutas Joshua D. Drake: > > In terms of idle time for gzip and other command to archive WAL offline, > > no difference in the environment was given other than the command to > > archive. My guess is because the user time is very large in gzip, it > >

Re: [HACKERS] Idle idea for a feature

2007-04-10 Thread NikhilS
Hi, On 4/11/07, Guillaume Smet <[EMAIL PROTECTED]> wrote: On 4/10/07, Tom Lane <[EMAIL PROTECTED]> wrote: > ISTM it'd be a good idea if it did, as "are there any incoming > foreign keys" seems to be a question we constantly ask when solving > update-performance problems, and there isn't any eas

Re: [HACKERS] Anyone interested in improving postgresql scaling?

2007-04-10 Thread Kris Kennaway
On Tue, Apr 10, 2007 at 03:52:00PM -0400, Tom Lane wrote: > Kris Kennaway <[EMAIL PROTECTED]> writes: > > On Tue, Apr 10, 2007 at 02:46:56PM -0400, Tom Lane wrote: > >> Oh, I'm sure the BSD kernel acts as you describe. But Mark's point is > >> that Postgres never has more than one process waiting

Re: [HACKERS] [EMAIL PROTECTED]: Re: Anyone interested in improving postgresql scaling?]

2007-04-10 Thread Kris Kennaway
On Tue, Apr 10, 2007 at 08:23:36PM -0400, Tom Lane wrote: > > I think the high number of setproctitle() calls are more problematic > > to us at the moment, Kris can comment on that. > > As of PG 8.2 it is possible to turn those off. I don't think there's a > lot of enthusiasm for turning them of

Re: [HACKERS] Anyone interested in improving postgresql scaling?

2007-04-10 Thread Kris Kennaway
On Tue, Apr 10, 2007 at 06:26:37PM -0400, Tom Lane wrote: > Kris Kennaway <[EMAIL PROTECTED]> writes: > > On Tue, Apr 10, 2007 at 05:36:17PM -0400, Tom Lane wrote: > >> Anyway I'd be interested to know what the test case is, and which PG > >> version you were testing. > > > I used 8.2 (and some ol

Re: [HACKERS] [EMAIL PROTECTED]: Re: Anyone interested in improving postgresql scaling?]

2007-04-10 Thread Kris Kennaway
On Wed, Apr 11, 2007 at 12:50:06PM +1200, Mark Kirkwood wrote: > Tom Lane wrote: > > > > >>I think the high number of setproctitle() calls are more problematic > >>to us at the moment, Kris can comment on that. > > > >As of PG 8.2 it is possible to turn those off. I don't think there's a > >lot o

Re: [HACKERS] [EMAIL PROTECTED]: Re: Anyone interested in improving postgresql scaling?]

2007-04-10 Thread Kris Kennaway
On Wed, Apr 11, 2007 at 01:03:50AM -0400, Tom Lane wrote: > Kris Kennaway <[EMAIL PROTECTED]> writes: > > I think the high number of setproctitle() calls are more problematic > > to us at the moment, Kris can comment on that. > > > Since we've basically had it handed to us that calling set

Re: [HACKERS] Anyone interested in improving postgresql scaling?

2007-04-10 Thread Kris Kennaway
On Tue, Apr 10, 2007 at 02:46:56PM -0400, Tom Lane wrote: > Kris Kennaway <[EMAIL PROTECTED]> writes: > >>> Make SYSV semaphores less dumb about process wakeups. Currently > >>> whenever the semaphore state changes, all processes sleeping on the > >>> semaphore are woken, even if we only have rele

Re: [HACKERS] Anyone interested in improving postgresql scaling?

2007-04-10 Thread Kris Kennaway
On Tue, Apr 10, 2007 at 05:36:17PM -0400, Tom Lane wrote: > Kris Kennaway <[EMAIL PROTECTED]> writes: > > I have not studied the exact code path, but there are indeed multiple > > wakeups happening from the semaphore code (as many as the number of > > active postgresql processes). It is easy to in

Re: [HACKERS] [EMAIL PROTECTED]: Re: Anyone interested in improving postgresql scaling?]

2007-04-10 Thread Tom Lane
Kris Kennaway <[EMAIL PROTECTED]> writes: > On Wed, Apr 11, 2007 at 01:03:50AM -0400, Tom Lane wrote: >> Well, the thing is, we've pretty much had it handed to us that >> current-command indicators that aren't up to date are not very useful. >> So rate-limited updates strike me as a useless comprom

[HACKERS] Question about SHM_QUEUE

2007-04-10 Thread ITAGAKI Takahiro
Hello, I have a question about SHM_QUEUE. Why do we need this component? We've already made some modules under the assumption that the base offset of shared memory is mapped to the same address for all processes. See comment in freespace.h: * Note: we handle pointers to these items as pointers,

Re: [HACKERS] prepared statements logging

2007-04-10 Thread tomas
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Tue, Apr 10, 2007 at 02:53:32AM -0700, marcofuics wrote: > Hi * > I am using the postgresql-8.2.3, with a jdbc-8.2-504 (the GeoNet > webServer tool...) My question is : > Is the <> server able to log the whole <> query? > {made by a prepared sta

Re: [HACKERS] CIC and deadlocks

2007-04-10 Thread Pavan Deolasee
On 4/1/07, Tom Lane <[EMAIL PROTECTED]> wrote: Good point. I'm envisioning a procarray.c function along the lines of bool TransactionHasSnapshot(xid) which returns true if the xid is currently listed in PGPROC and has a nonzero xmin. CIC's cleanup wait loop would check this and ignore

Re: [HACKERS] Question about SHM_QUEUE

2007-04-10 Thread Tom Lane
ITAGAKI Takahiro <[EMAIL PROTECTED]> writes: > I have a question about SHM_QUEUE. Why do we need this component? It's a hangover from Berkeley days that no one has felt a need to remove yet. The convention back then was that shared memory might be mapped to different addresses in different proces

Re: [HACKERS] [DOCS] uuid type not documented

2007-04-10 Thread Marko Kreen
On 4/10/07, Peter Eisentraut <[EMAIL PROTECTED]> wrote: The problem is that most of the standard methods are platform dependent, as they require MAC addresses or a "good" random source, for instance. FYI: good random source is already available in pgcrypto, it uses either OpenSSL RAND_bytes() o

Re: [HACKERS] CIC and deadlocks

2007-04-10 Thread Tom Lane
"Pavan Deolasee" <[EMAIL PROTECTED]> writes: > When I looked at the code, it occurred to me that possibly we are > OK with just taking shared lock on the procarray. That means that > some other transaction can concurrently set its serializable snapshot > while we are scanning the procarray. But tha