Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-06-04 Thread Bruce Momjian
Later version of this patch added to the patch queue. Your patch has been added to the PostgreSQL unapplied patches list at: http://momjian.postgresql.org/cgi-bin/pgpatches It will be applied as soon as one of the PostgreSQL committers reviews and approves it.

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-07 Thread Marc G. Fournier
On Fri, 7 Jan 2005, Bruce Momjian wrote: Do we want to add this additional log infor to CVS for 8.0? No, unless we're looking for an RC5? --- Simon Riggs wrote: On Mon, 2005-01-03 at 19:14 -0500, Bruce Momjian wrote: Simon

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-07 Thread Tom Lane
Marc G. Fournier [EMAIL PROTECTED] writes: On Fri, 7 Jan 2005, Bruce Momjian wrote: Do we want to add this additional log infor to CVS for 8.0? No, unless we're looking for an RC5? I vote no as well. While it's probably not a dangerous change, the need for it has not been demonstrated.

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-07 Thread Bruce Momjian
Tom Lane wrote: Marc G. Fournier [EMAIL PROTECTED] writes: On Fri, 7 Jan 2005, Bruce Momjian wrote: Do we want to add this additional log infor to CVS for 8.0? No, unless we're looking for an RC5? I vote no as well. While it's probably not a dangerous change, the need for it has not

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-06 Thread Bruce Momjian
Do we want to add this additional log infor to CVS for 8.0? --- Simon Riggs wrote: On Mon, 2005-01-03 at 19:14 -0500, Bruce Momjian wrote: Simon Riggs wrote: Here's my bgwriter instrumentation patch, which gives info

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-04 Thread Simon Riggs
On Mon, 2005-01-03 at 19:14 -0500, Bruce Momjian wrote: Simon Riggs wrote: Here's my bgwriter instrumentation patch, which gives info that could allow the bgwriter settings to be tuned. Uh, what does this do exactly? Add additional logging output? Produces output like this...

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-03 Thread Bruce Momjian
OK, we have a submitted patch that attempts to improve bgwriter by making bgwriter_percent control what percentage of the buffer is scanned. The patch still needs doc changes and a change to the default value but at this point we need a vote on the patch. Is it: * too late for 8.0

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-03 Thread Tom Lane
Bruce Momjian pgman@candle.pha.pa.us writes: OK, we have a submitted patch that attempts to improve bgwriter by making bgwriter_percent control what percentage of the buffer is scanned. The patch still needs doc changes and a change to the default value but at this point we need a vote on

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-03 Thread Marc G. Fournier
On Mon, 3 Jan 2005, Bruce Momjian wrote: OK, we have a submitted patch that attempts to improve bgwriter by making bgwriter_percent control what percentage of the buffer is scanned. The patch still needs doc changes and a change to the default value but at this point we need a vote on the patch.

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-03 Thread Simon Riggs
On Mon, 2005-01-03 at 20:09, Bruce Momjian wrote: OK, we have a submitted patch that attempts to improve bgwriter by making bgwriter_percent control what percentage of the buffer is scanned. The patch still needs doc changes and a change to the default value but at this point we need a vote

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-03 Thread Bruce Momjian
Simon Riggs wrote: On Mon, 2005-01-03 at 20:09, Bruce Momjian wrote: OK, we have a submitted patch that attempts to improve bgwriter by making bgwriter_percent control what percentage of the buffer is scanned. The patch still needs doc changes and a change to the default value but at

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-03 Thread Simon Riggs
On Mon, 2005-01-03 at 23:03, Bruce Momjian wrote: Simon Riggs wrote: On Mon, 2005-01-03 at 20:09, Bruce Momjian wrote: OK, we have a submitted patch that attempts to improve bgwriter by making bgwriter_percent control what percentage of the buffer is scanned. The patch still

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-03 Thread Bruce Momjian
Simon Riggs wrote: Here's my bgwriter instrumentation patch, which gives info that could allow the bgwriter settings to be tuned. Uh, what does this do exactly? Add additional logging output? -- Bruce Momjian| http://candle.pha.pa.us pgman@candle.pha.pa.us

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-03 Thread Bruce Momjian
This has been saved for the 8.1 release: http:/momjian.postgresql.org/cgi-bin/pgpatches2 --- Simon Riggs wrote: On Sat, 2005-01-01 at 17:47, Simon Riggs wrote: On Sat, 2005-01-01 at 17:01, Bruce Momjian wrote:

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-01 Thread Simon Riggs
On Sat, 2005-01-01 at 06:20, Bruce Momjian wrote: This change isn't going to make it for RC3, and it probably not something we want to rush. OK. Thank you. I think there are a few issues involved: o everyone agrees the current meaning of bgwriter_percent is useless

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-01 Thread Bruce Momjian
Simon Riggs wrote: On Sat, 2005-01-01 at 06:20, Bruce Momjian wrote: This change isn't going to make it for RC3, and it probably not something we want to rush. OK. Thank you. I think there are a few issues involved: o everyone agrees the current meaning of bgwriter_percent is

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-01 Thread Simon Riggs
On Sat, 2005-01-01 at 17:01, Bruce Momjian wrote: Simon Riggs wrote: Well, I think we're saying: its not in 8.0 now, and we take our time to consider patches for 8.1 and accept the situation that the parameter names/meaning will change in next release. I have no problem doing

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-01 Thread Tom Lane
Bruce Momjian pgman@candle.pha.pa.us writes: o everyone agrees the current meaning of bgwriter_percent is useless (percent of dirty buffers) Oh? It's not useless by any means; it's a perfectly reasonable and useful definition that happens to be expensive to implement. One of

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-01 Thread Simon Riggs
On Sat, 2005-01-01 at 17:47, Simon Riggs wrote: On Sat, 2005-01-01 at 17:01, Bruce Momjian wrote: Simon Riggs wrote: Well, I think we're saying: its not in 8.0 now, and we take our time to consider patches for 8.1 and accept the situation that the parameter names/meaning will

Re: [PATCHES] [HACKERS] Bgwriter behavior

2005-01-01 Thread Bruce Momjian
Tom Lane wrote: Bruce Momjian pgman@candle.pha.pa.us writes: o everyone agrees the current meaning of bgwriter_percent is useless (percent of dirty buffers) Oh? It's not useless by any means; it's a perfectly reasonable and useful definition that happens to be expensive to

Re: [HACKERS] Bgwriter behavior

2004-12-31 Thread Simon Riggs
On Fri, 2004-12-31 at 01:14, Bruce Momjian wrote: Simon Riggs wrote: On Mon, 2004-12-27 at 22:21, Bruce Momjian wrote: Should we consider at least adjusting the meaning of bgwriter_percent? Yes. As things stand, this is the only change that seems safe. Here's a very short patch

Re: [PATCHES] [HACKERS] Bgwriter behavior

2004-12-31 Thread Bruce Momjian
This change isn't going to make it for RC3, and it probably not something we want to rush. I think there are a few issues involved: o everyone agrees the current meaning of bgwriter_percent is useless (percent of dirty buffers) o removal of bgwriter_percent will

Re: [HACKERS] Bgwriter behavior

2004-12-30 Thread Simon Riggs
On Mon, 2004-12-27 at 22:21, Bruce Momjian wrote: Should we consider at least adjusting the meaning of bgwriter_percent? Yes. As things stand, this is the only change that seems safe. Here's a very short patch that implements this change within BufferSync in bufmgr.c - No algorithm changes -

Re: [HACKERS] Bgwriter behavior

2004-12-30 Thread Bruce Momjian
Simon Riggs wrote: On Mon, 2004-12-27 at 22:21, Bruce Momjian wrote: Should we consider at least adjusting the meaning of bgwriter_percent? Yes. As things stand, this is the only change that seems safe. Here's a very short patch that implements this change within BufferSync in bufmgr.c

Re: [HACKERS] Bgwriter behavior

2004-12-29 Thread Manfred Koizar
[I know I'm late and this has already been discussed by Richrad, Tom, et al., but ...] On Tue, 21 Dec 2004 16:17:17 -0600, Jim C. Nasby [EMAIL PROTECTED] wrote: look at where the last page you wrote out has ended up in the LRU list since you last ran, and start scanning from there (by definition

Re: [HACKERS] Bgwriter behavior

2004-12-28 Thread Bruce Momjian
Added to TODO: * Improve the background writer Allow the background writer to more efficiently write dirty buffers from the end of the LRU cache and use a clock sweep algorithm to write other dirty buffers to reduced checkpoint I/O

Re: [HACKERS] Bgwriter behavior

2004-12-28 Thread Bruce Momjian
John Hansen wrote: I ran some tests last week and can report results similar on Tom's test: pgbench -i -s 10 bench pgbench -c 10 -t 1 bench The tests were on a machine with a single SCSI drive that doesn't lie about fsync. I found 7.4.X got around 75tps while 8.0 got

Re: [HACKERS] Bgwriter behavior

2004-12-28 Thread Tom Lane
Bruce Momjian pgman@candle.pha.pa.us writes: John Hansen wrote: On another note, how do you know for sure, that your drive does not lie about fsync? I just tried and got 115tps with fsync off vs 100 with fsync on, so fsync is certainly doing something. [ raised eyebrow... ] Something is

Re: [HACKERS] Bgwriter behavior

2004-12-28 Thread Bruce Momjian
Tom Lane wrote: Bruce Momjian pgman@candle.pha.pa.us writes: John Hansen wrote: On another note, how do you know for sure, that your drive does not lie about fsync? I just tried and got 115tps with fsync off vs 100 with fsync on, so fsync is certainly doing something. [ raised

Re: [HACKERS] Bgwriter behavior

2004-12-28 Thread Simon Riggs
On Tue, 2004-12-28 at 07:23, John Hansen wrote: I ran some tests last week and can report results similar on Tom's test: pgbench -i -s 10 bench pgbench -c 10 -t 1 bench The tests were on a machine with a single SCSI drive that doesn't lie about fsync. I found 7.4.X got

Re: [HACKERS] Bgwriter behavior

2004-12-28 Thread John Hansen
I ran some tests last week and can report results similar on Tom's test: pgbench -i -s 10 bench pgbench -c 10 -t 1 bench don't you have to specify the scaling factor for the benchmark as well? as in pgbench -c 10 -t 1 -s 10 bench ? I just tried and got 115tps with

Re: [HACKERS] Bgwriter behavior

2004-12-28 Thread Bruce Momjian
John Hansen wrote: I ran some tests last week and can report results similar on Tom's test: pgbench -i -s 10 bench pgbench -c 10 -t 1 bench don't you have to specify the scaling factor for the benchmark as well? as in pgbench -c 10 -t 1 -s 10 bench

Re: [HACKERS] Bgwriter behavior

2004-12-28 Thread Mark Kirkwood
Bruce Momjian wrote: well, I usually get results that differ by that much from run to run. Probably you ran in to more checkpoints on the second test. Also, did you reinitialize the bench database with pgbench -i ? I destroyed the database and recreated it. The only way I managed to

Re: [HACKERS] Bgwriter behavior

2004-12-27 Thread Bruce Momjian
Simon Riggs wrote: On Wed, 2004-12-22 at 04:43, Tom Lane wrote: Bruce Momjian pgman@candle.pha.pa.us writes: So what are we doing for 8.0? Well, it looks like RC2 has already crashed and burned --- I can't imagine that Marc will let us release without an RC3 given what was committed

Re: [HACKERS] Bgwriter behavior

2004-12-27 Thread John Hansen
I ran some tests last week and can report results similar on Tom's test: pgbench -i -s 10 bench pgbench -c 10 -t 1 bench The tests were on a machine with a single SCSI drive that doesn't lie about fsync. I found 7.4.X got around 75tps while 8.0 got 100tps, very similar

Re: [HACKERS] Bgwriter behavior

2004-12-23 Thread Tom Lane
Bruce Momjian pgman@candle.pha.pa.us writes: I remember the other difference between 8.0 and pre-8.0. When a backend has to write a block in 8.0, it does a write _plus_ fsync(), while in pre-8.0 it did only a write. There was a proposal to pass backend write information to the background

Re: [HACKERS] Bgwriter behavior

2004-12-23 Thread Bruce Momjian
Tom Lane wrote: Bruce Momjian pgman@candle.pha.pa.us writes: I remember the other difference between 8.0 and pre-8.0. When a backend has to write a block in 8.0, it does a write _plus_ fsync(), while in pre-8.0 it did only a write. There was a proposal to pass backend write information

Re: [HACKERS] Bgwriter behavior

2004-12-23 Thread Simon Riggs
On Wed, 2004-12-22 at 04:43, Tom Lane wrote: Bruce Momjian pgman@candle.pha.pa.us writes: So what are we doing for 8.0? Well, it looks like RC2 has already crashed and burned --- I can't imagine that Marc will let us release without an RC3 given what was committed today, never mind the

Re: [HACKERS] Bgwriter behavior

2004-12-22 Thread Bruce Momjian
Tom Lane wrote: Bruce Momjian pgman@candle.pha.pa.us writes: So what are we doing for 8.0? Well, it looks like RC2 has already crashed and burned --- I can't imagine that Marc will let us release without an RC3 given what was committed today, never mind the btree bug that Mark Wong seems

[HACKERS] Bgwriter behavior

2004-12-21 Thread Bruce Momjian
Tom Lane wrote: Gavin Sherry [EMAIL PROTECTED] writes: I was also thinking of benchmarking the effect of changing the algorithm in StrategyDirtyBufferList(): currently, for each iteration of the loop we read a buffer from each of T1 and T2. I was wondering what effect reading T1 first

Re: [HACKERS] Bgwriter behavior

2004-12-21 Thread Tom Lane
Bruce Momjian [EMAIL PROTECTED] writes: First, we remove the GUC bgwriter_maxpages because I don't see a good way to set a default for that. A default value needs to be based on a percentage of the full buffer cache size. This is nonsense. The admin knows what he set shared_buffers to, and

Re: [HACKERS] Bgwriter behavior

2004-12-21 Thread Jim C. Nasby
A quick $0.02 on how DB2 does this (at least in 7.x). They used a combination of everything that's been discussed. The first priority of their background writer was to keep the LRU end of the cache free so individual backends would never have to wait to get a page. Then, they would look to pages

Re: [HACKERS] Bgwriter behavior

2004-12-21 Thread Bruce Momjian
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: First, we remove the GUC bgwriter_maxpages because I don't see a good way to set a default for that. A default value needs to be based on a percentage of the full buffer cache size. This is nonsense. The admin knows what he set

Re: [HACKERS] Bgwriter behavior

2004-12-21 Thread Tom Lane
Bruce Momjian [EMAIL PROTECTED] writes: So what are we doing for 8.0? Well, it looks like RC2 has already crashed and burned --- I can't imagine that Marc will let us release without an RC3 given what was committed today, never mind the btree bug that Mark Wong seems to have found. So maybe we

Re: [HACKERS] Bgwriter behavior

2004-12-21 Thread Joshua D. Drake
At the moment my inclination is to sit on what we have. I've not seen any indication that 8.0 is really worse than earlier releases; the most you could argue against it is that it's not as much better as we hoped. That's not grounds to muck around at the RC3 stage. If is is any help, CMD is

Re: [HACKERS] Bgwriter behavior

2004-12-21 Thread Bruce Momjian
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: So what are we doing for 8.0? Well, it looks like RC2 has already crashed and burned --- I can't imagine that Marc will let us release without an RC3 given what was committed today, never mind the btree bug that Mark Wong seems to

Re: [HACKERS] Bgwriter behavior

2004-12-21 Thread Tom Lane
Bruce Momjian [EMAIL PROTECTED] writes: The only way I could see it being worse than pre-8.0 is that the bgwriter is doing fsync of all open files rather than using sync. Other than that, I think it should behave the same, or slightly better, right? It's possible that there exist platforms