Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-06-04 Thread Bruce Momjian
Later version of this patch added to the patch queue. Your patch has been added to the PostgreSQL unapplied patches list at: http://momjian.postgresql.org/cgi-bin/pgpatches It will be applied as soon as one of the PostgreSQL committers reviews and approves it. -

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-07 Thread Bruce Momjian
Tom Lane wrote: > "Marc G. Fournier" <[EMAIL PROTECTED]> writes: > > On Fri, 7 Jan 2005, Bruce Momjian wrote: > >> Do we want to add this additional log infor to CVS for 8.0? > > > No, unless we're looking for an RC5? > > I vote no as well. While it's probably not a dangerous change, the need >

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-07 Thread Tom Lane
"Marc G. Fournier" <[EMAIL PROTECTED]> writes: > On Fri, 7 Jan 2005, Bruce Momjian wrote: >> Do we want to add this additional log infor to CVS for 8.0? > No, unless we're looking for an RC5? I vote no as well. While it's probably not a dangerous change, the need for it has not been demonstrated

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-07 Thread Marc G. Fournier
On Fri, 7 Jan 2005, Bruce Momjian wrote: Do we want to add this additional log infor to CVS for 8.0? No, unless we're looking for an RC5? --- Simon Riggs wrote: On Mon, 2005-01-03 at 19:14 -0500, Bruce Momjian wrote: Simon Rig

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-06 Thread Bruce Momjian
Do we want to add this additional log infor to CVS for 8.0? --- Simon Riggs wrote: > On Mon, 2005-01-03 at 19:14 -0500, Bruce Momjian wrote: > > Simon Riggs wrote: > > > Here's my bgwriter instrumentation patch, which gives

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-04 Thread Simon Riggs
On Mon, 2005-01-03 at 19:14 -0500, Bruce Momjian wrote: > Simon Riggs wrote: > > Here's my bgwriter instrumentation patch, which gives info that could > > allow the bgwriter settings to be tuned. > > Uh, what does this do exactly? Add additional logging output? > Produces output like this... D

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-03 Thread Bruce Momjian
This has been saved for the 8.1 release: http:/momjian.postgresql.org/cgi-bin/pgpatches2 --- Simon Riggs wrote: > On Sat, 2005-01-01 at 17:47, Simon Riggs wrote: > > On Sat, 2005-01-01 at 17:01, Bruce Momjian wrote:

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-03 Thread Bruce Momjian
Simon Riggs wrote: > Here's my bgwriter instrumentation patch, which gives info that could > allow the bgwriter settings to be tuned. Uh, what does this do exactly? Add additional logging output? -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-03 Thread Simon Riggs
On Mon, 2005-01-03 at 23:03, Bruce Momjian wrote: > Simon Riggs wrote: > > On Mon, 2005-01-03 at 20:09, Bruce Momjian wrote: > > > OK, we have a submitted patch that attempts to improve bgwriter by > > > making bgwriter_percent control what percentage of the buffer is > > > scanned. > > > > > > Th

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-03 Thread Bruce Momjian
Simon Riggs wrote: > On Mon, 2005-01-03 at 20:09, Bruce Momjian wrote: > > OK, we have a submitted patch that attempts to improve bgwriter by > > making bgwriter_percent control what percentage of the buffer is > > scanned. > > > > The patch still needs doc changes and a change to the default valu

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-03 Thread Simon Riggs
On Mon, 2005-01-03 at 20:09, Bruce Momjian wrote: > OK, we have a submitted patch that attempts to improve bgwriter by > making bgwriter_percent control what percentage of the buffer is > scanned. > > The patch still needs doc changes and a change to the default value but > at this point we need a

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-03 Thread Marc G. Fournier
On Mon, 3 Jan 2005, Bruce Momjian wrote: OK, we have a submitted patch that attempts to improve bgwriter by making bgwriter_percent control what percentage of the buffer is scanned. The patch still needs doc changes and a change to the default value but at this point we need a vote on the patch. I

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-03 Thread Tom Lane
Bruce Momjian writes: > OK, we have a submitted patch that attempts to improve bgwriter by > making bgwriter_percent control what percentage of the buffer is > scanned. > The patch still needs doc changes and a change to the default value but > at this point we need a vote on the patch. Is it:

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-03 Thread Bruce Momjian
OK, we have a submitted patch that attempts to improve bgwriter by making bgwriter_percent control what percentage of the buffer is scanned. The patch still needs doc changes and a change to the default value but at this point we need a vote on the patch. Is it: * too late for 8.0

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-01 Thread Bruce Momjian
Tom Lane wrote: > Bruce Momjian writes: > > o everyone agrees the current meaning of bgwriter_percent is > >useless (percent of dirty buffers) > > Oh? > > It's not useless by any means; it's a perfectly reasonable and useful > definition that happens to be expensive to implement. O

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-01 Thread Simon Riggs
On Sat, 2005-01-01 at 17:47, Simon Riggs wrote: > On Sat, 2005-01-01 at 17:01, Bruce Momjian wrote: > > Simon Riggs wrote: > > > > > Well, I think we're saying: its not in 8.0 now, and we take our time to > > > consider patches for 8.1 and accept the situation that the parameter > > > names/meani

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-01 Thread Tom Lane
Bruce Momjian writes: > o everyone agrees the current meaning of bgwriter_percent is > useless (percent of dirty buffers) Oh? It's not useless by any means; it's a perfectly reasonable and useful definition that happens to be expensive to implement. One of the questions that is

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-01 Thread Simon Riggs
On Sat, 2005-01-01 at 17:01, Bruce Momjian wrote: > Simon Riggs wrote: > > > Well, I think we're saying: its not in 8.0 now, and we take our time to > > consider patches for 8.1 and accept the situation that the parameter > > names/meaning will change in next release. > > I have no problem doing

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-01 Thread Bruce Momjian
Simon Riggs wrote: > On Sat, 2005-01-01 at 06:20, Bruce Momjian wrote: > > This change isn't going to make it for RC3, and it probably not > > something we want to rush. > > OK. Thank you. > > > I think there are a few issues involved: > > > > o everyone agrees the current meaning of bgwrit

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-01 Thread Simon Riggs
On Sat, 2005-01-01 at 06:20, Bruce Momjian wrote: > This change isn't going to make it for RC3, and it probably not > something we want to rush. OK. Thank you. > I think there are a few issues involved: > > o everyone agrees the current meaning of bgwriter_percent is > useless (p

Re: [PATCHES] [HACKERS] Bgwriter behavior

2004-12-31 Thread Bruce Momjian
This change isn't going to make it for RC3, and it probably not something we want to rush. I think there are a few issues involved: o everyone agrees the current meaning of bgwriter_percent is useless (percent of dirty buffers) o removal of bgwriter_percent will caus

Re: [HACKERS] Bgwriter behavior

2004-12-31 Thread Simon Riggs
On Fri, 2004-12-31 at 01:14, Bruce Momjian wrote: > Simon Riggs wrote: > > On Mon, 2004-12-27 at 22:21, Bruce Momjian wrote: > > > Should we consider at least adjusting the meaning of bgwriter_percent? > > > > Yes. As things stand, this is the only change that seems safe. > > > > Here's a very sh

Re: [HACKERS] Bgwriter behavior

2004-12-30 Thread Bruce Momjian
Simon Riggs wrote: > On Mon, 2004-12-27 at 22:21, Bruce Momjian wrote: > > Should we consider at least adjusting the meaning of bgwriter_percent? > > Yes. As things stand, this is the only change that seems safe. > > Here's a very short patch that implements this change within BufferSync > in buf

Re: [HACKERS] Bgwriter behavior

2004-12-30 Thread Simon Riggs
On Mon, 2004-12-27 at 22:21, Bruce Momjian wrote: > Should we consider at least adjusting the meaning of bgwriter_percent? Yes. As things stand, this is the only change that seems safe. Here's a very short patch that implements this change within BufferSync in bufmgr.c - No algorithm changes -

Re: [HACKERS] Bgwriter behavior

2004-12-29 Thread Manfred Koizar
[I know I'm late and this has already been discussed by Richrad, Tom, et al., but ...] On Tue, 21 Dec 2004 16:17:17 -0600, "Jim C. Nasby" <[EMAIL PROTECTED]> wrote: >look at where the last page you wrote out has ended up in the LRU list >since you last ran, and start scanning from there (by defini

Re: [HACKERS] Bgwriter behavior

2004-12-28 Thread Mark Kirkwood
Bruce Momjian wrote: well, I usually get results that differ by that much from run to run. Probably you ran in to more checkpoints on the second test. Also, did you reinitialize the bench database with pgbench -i ? I destroyed the database and recreated it. The only way I managed to contro

Re: [HACKERS] Bgwriter behavior

2004-12-28 Thread Bruce Momjian
John Hansen wrote: > > > > I ran some tests last week and can report results similar on Tom's test: > > > > > > > > pgbench -i -s 10 bench > > > > pgbench -c 10 -t 1 bench > > > > > > don't you have to specify the scaling factor for the benchmark as well? > as in pgbench -c 1

Re: [HACKERS] Bgwriter behavior

2004-12-28 Thread John Hansen
> > > I ran some tests last week and can report results similar on Tom's test: > > > > > > pgbench -i -s 10 bench > > > pgbench -c 10 -t 1 bench > > > don't you have to specify the scaling factor for the benchmark as well? as in pgbench -c 10 -t 1 -s 10 bench ? > I just tried and go

Re: [HACKERS] Bgwriter behavior

2004-12-28 Thread Simon Riggs
On Tue, 2004-12-28 at 07:23, John Hansen wrote: > > I ran some tests last week and can report results similar on Tom's test: > > > > pgbench -i -s 10 bench > > pgbench -c 10 -t 1 bench > > > > The tests were on a machine with a single SCSI drive that doesn't lie > > about fsync. I fo

Re: [HACKERS] Bgwriter behavior

2004-12-28 Thread Bruce Momjian
Tom Lane wrote: > Bruce Momjian writes: > > John Hansen wrote: > >> On another note, how do you know for sure, that your drive does not lie > >> about fsync? > > > I just tried and got 115tps with fsync off vs 100 with fsync on, so > > fsync is certainly doing something. > > [ raised eyebrow...

Re: [HACKERS] Bgwriter behavior

2004-12-28 Thread Tom Lane
Bruce Momjian writes: > John Hansen wrote: >> On another note, how do you know for sure, that your drive does not lie >> about fsync? > I just tried and got 115tps with fsync off vs 100 with fsync on, so > fsync is certainly doing something. [ raised eyebrow... ] Something is wrong with that.

Re: [HACKERS] Bgwriter behavior

2004-12-28 Thread Bruce Momjian
John Hansen wrote: > > I ran some tests last week and can report results similar on Tom's test: > > > > pgbench -i -s 10 bench > > pgbench -c 10 -t 1 bench > > > > The tests were on a machine with a single SCSI drive that doesn't lie > > about fsync. I found 7.4.X got around 75tps wh

Re: [HACKERS] Bgwriter behavior

2004-12-28 Thread Bruce Momjian
Added to TODO: * Improve the background writer Allow the background writer to more efficiently write dirty buffers from the end of the LRU cache and use a clock sweep algorithm to write other dirty buffers to reduced checkpoint I/O

Re: [HACKERS] Bgwriter behavior

2004-12-27 Thread John Hansen
> I ran some tests last week and can report results similar on Tom's test: > > pgbench -i -s 10 bench > pgbench -c 10 -t 1 bench > > The tests were on a machine with a single SCSI drive that doesn't lie > about fsync. I found 7.4.X got around 75tps while 8.0 got 100tps, very > si

Re: [HACKERS] Bgwriter behavior

2004-12-27 Thread Bruce Momjian
Simon Riggs wrote: > On Wed, 2004-12-22 at 04:43, Tom Lane wrote: > > Bruce Momjian writes: > > > So what are we doing for 8.0? > > > > Well, it looks like RC2 has already crashed and burned --- I can't > > imagine that Marc will let us release without an RC3 given what was > > committed today, n

Re: [HACKERS] Bgwriter behavior

2004-12-23 Thread Simon Riggs
On Wed, 2004-12-22 at 04:43, Tom Lane wrote: > Bruce Momjian writes: > > So what are we doing for 8.0? > > Well, it looks like RC2 has already crashed and burned --- I can't > imagine that Marc will let us release without an RC3 given what was > committed today, never mind the btree bug that Mark

Re: [HACKERS] Bgwriter behavior

2004-12-23 Thread Bruce Momjian
Tom Lane wrote: > Bruce Momjian writes: > > I remember the other difference between 8.0 and pre-8.0. When a backend > > has to write a block in 8.0, it does a write _plus_ fsync(), while in > > pre-8.0 it did only a write. There was a proposal to pass backend write > > information to the backgro

Re: [HACKERS] Bgwriter behavior

2004-12-23 Thread Tom Lane
Bruce Momjian writes: > I remember the other difference between 8.0 and pre-8.0. When a backend > has to write a block in 8.0, it does a write _plus_ fsync(), while in > pre-8.0 it did only a write. There was a proposal to pass backend write > information to the background writer so it would kno

Re: [HACKERS] Bgwriter behavior

2004-12-22 Thread Bruce Momjian
Tom Lane wrote: > Bruce Momjian writes: > > So what are we doing for 8.0? > > Well, it looks like RC2 has already crashed and burned --- I can't > imagine that Marc will let us release without an RC3 given what was > committed today, never mind the btree bug that Mark Wong seems to have > found.

Re: [HACKERS] Bgwriter behavior

2004-12-21 Thread Tom Lane
Bruce Momjian <[EMAIL PROTECTED]> writes: > The only way I could see it being worse than pre-8.0 is that the > bgwriter is doing fsync of all open files rather than using sync. Other > than that, I think it should behave the same, or slightly better, > right? It's possible that there exist platfo

Re: [HACKERS] Bgwriter behavior

2004-12-21 Thread Bruce Momjian
Tom Lane wrote: > Bruce Momjian <[EMAIL PROTECTED]> writes: > > So what are we doing for 8.0? > > Well, it looks like RC2 has already crashed and burned --- I can't > imagine that Marc will let us release without an RC3 given what was > committed today, never mind the btree bug that Mark Wong seem

Re: [HACKERS] Bgwriter behavior

2004-12-21 Thread Joshua D. Drake
At the moment my inclination is to sit on what we have. I've not seen any indication that 8.0 is really worse than earlier releases; the most you could argue against it is that it's not as much better as we hoped. That's not grounds to muck around at the RC3 stage. If is is any help, CMD is ba

Re: [HACKERS] Bgwriter behavior

2004-12-21 Thread Tom Lane
Bruce Momjian <[EMAIL PROTECTED]> writes: > So what are we doing for 8.0? Well, it looks like RC2 has already crashed and burned --- I can't imagine that Marc will let us release without an RC3 given what was committed today, never mind the btree bug that Mark Wong seems to have found. So maybe w

Re: [HACKERS] Bgwriter behavior

2004-12-21 Thread Bruce Momjian
Tom Lane wrote: > Bruce Momjian <[EMAIL PROTECTED]> writes: > > First, we remove the GUC bgwriter_maxpages because I don't see a good > > way to set a default for that. A default value needs to be based on a > > percentage of the full buffer cache size. > > This is nonsense. The admin knows what

Re: [HACKERS] Bgwriter behavior

2004-12-21 Thread Jim C. Nasby
A quick $0.02 on how DB2 does this (at least in 7.x). They used a combination of everything that's been discussed. The first priority of their background writer was to keep the LRU end of the cache free so individual backends would never have to wait to get a page. Then, they would look to pages t

Re: [HACKERS] Bgwriter behavior

2004-12-21 Thread Tom Lane
Bruce Momjian <[EMAIL PROTECTED]> writes: > First, we remove the GUC bgwriter_maxpages because I don't see a good > way to set a default for that. A default value needs to be based on a > percentage of the full buffer cache size. This is nonsense. The admin knows what he set shared_buffers to, a

[HACKERS] Bgwriter behavior

2004-12-21 Thread Bruce Momjian
Tom Lane wrote: > Gavin Sherry <[EMAIL PROTECTED]> writes: > > I was also thinking of benchmarking the effect of changing the algorithm > > in StrategyDirtyBufferList(): currently, for each iteration of the loop we > > read a buffer from each of T1 and T2. I was wondering what effect reading > > T1