Re: [PATCHES] Load distributed checkpoint V3
On Mon, 26 Mar 2007, ITAGAKI Takahiro wrote: I'm assuming checkpoints are triggered by timeout in normal use -- and it's my recommended configuration whether the patch is installed or not. I'm curious what other people running fairly serious hardware do in this area for write-heavy loads, whether it's timeout or segment limits that normally trigger their checkpoints. I'm testing on a slightly different class of machine than your sample results, something that is in the 1500 TPS range running the pgbench test you describe. Running that test, I always hit the checkpoint_segments wall well before any reasonable timeout. With 64 segments, I get a checkpoint every two minutes or so. There's something I'm working on this week that may help out other people trying to test your patch out. I've put together some simple scripts that graph (patched) pgbench results, which make it very easy to see what changes when you alter the checkpoint behavior. Edges are still rough but the scripts work for me, will be polishing and testing over the next few days: http://www.westnet.com/~gsmith/content/postgresql/pgbench.htm (Note that the example graphs there aren't from the production system I mentioned above, they're from my server at home, which is similar to the system your results came from). -- * Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [PATCHES] Load distributed checkpoint V3
Greg Smith <[EMAIL PROTECTED]> wrote: > > Here is the latest version of Load distributed checkpoint patch. > > Couple of questions for you: > > -Is it still possible to get the original behavior by adjusting your > tunables? It would be nice to do a before/after without having to > recompile, and I know I'd be concerned about something so different > becoming the new default behavior. Yes, if you want the original behavior, please set all of checkpoint_[write|nap|sync]_percent to zero. They can be changed at SIGHUP timing (pg_ctl reload). The new default configurations are write/nap/sync = 50%/10%/20%. There might be room for discussion in choice of the values. > -Can you suggest a current test case to demonstrate the performance > improvement here? I've tried several variations on stretching out > checkpoints like you're doing here and they all made slow checkpoint > issues even worse on my Linux system. I'm trying to evaluate this fairly. You might need to increase checkpoint_segments and checkpoint_timeout. Here is the results on my machine: http://archives.postgresql.org/pgsql-hackers/2007-02/msg01613.php I've set the values to 32 segs and 15 min to take advantage of it in the case of pgbench -s100 then. > -This code operates on the assumption you have a good value for the > checkpoint timeout. Have you tested its behavior when checkpoints are > being triggered by checkpoint_segments being reached instead? This patch does not work fully when checkpoints are triggered by segments. Write phases still work because they refer to consumption of segments, but nap and fsync phases only check amount of time. I'm assuming checkpoints are triggered by timeout in normal use -- and it's my recommended configuration whether the patch is installed or not. Regards, --- ITAGAKI Takahiro NTT Open Source Software Center ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [PATCHES] Improvement of procArray.xmin for VACUUM
"Tom Lane" <[EMAIL PROTECTED]> writes: > Gregory Stark <[EMAIL PROTECTED]> writes: >> Well I think this would be the same infrastructure we would need to do >> the other discussed improvement to address pg_dump's impact. That >> would require us to publish the youngest xmax of the live >> snapshots. Vacuum could deduce that that xid cannot possibly see any >> transactions between the youngest extant xmax and the oldest >> in-progress transaction. > > ... and do what with the knowledge? Not remove tuples, because any such > tuples would be part of RECENTLY_DEAD update chains that that xact might > be following now or in the future. Well that just means it might require extra work, not that it would be impossible. Firstly, some tuples would not be part of a chain and could be cleaned up anyways. Others would be part of a HOT chain which might make it easier to clean them up. But even for tuples that are part of a chain there may be solutions. We could truncate the tuple to just the MVCC information so subsequent transactions can find the head. Or we might be able to go back and edit the forward link to skip the dead intermediate tuples (and somehow deal with the race conditions...) -- Gregory Stark EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [PATCHES] Improvement of procArray.xmin for VACUUM
Gregory Stark <[EMAIL PROTECTED]> writes: > Well I think this would be the same infrastructure we would need to do > the other discussed improvement to address pg_dump's impact. That > would require us to publish the youngest xmax of the live > snapshots. Vacuum could deduce that that xid cannot possibly see any > transactions between the youngest extant xmax and the oldest > in-progress transaction. ... and do what with the knowledge? Not remove tuples, because any such tuples would be part of RECENTLY_DEAD update chains that that xact might be following now or in the future. regards, tom lane ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [PATCHES] Load distributed checkpoint V3
On Fri, 23 Mar 2007, ITAGAKI Takahiro wrote: Here is the latest version of Load distributed checkpoint patch. Couple of questions for you: -Is it still possible to get the original behavior by adjusting your tunables? It would be nice to do a before/after without having to recompile, and I know I'd be concerned about something so different becoming the new default behavior. -Can you suggest a current test case to demonstrate the performance improvement here? I've tried several variations on stretching out checkpoints like you're doing here and they all made slow checkpoint issues even worse on my Linux system. I'm trying to evaluate this fairly. -This code operates on the assumption you have a good value for the checkpoint timeout. Have you tested its behavior when checkpoints are being triggered by checkpoint_segments being reached instead? -- * Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
