I agree that 30k looks like the magic delay, and probably 30/5 would be a
good conservative choice. But now I think about the choice of number, I
think it must vary with the speed of the machine and length of the
transactions; at 20tps, each TX is completing in around 50ms.
I think disk
At 10:56 27/02/01 +0100, Zeugswetter Andreas SB wrote:
I agree that 30k looks like the magic delay, and probably 30/5 would be a
good conservative choice. But now I think about the choice of number, I
think it must vary with the speed of the machine and length of the
transactions; at
One thing that I remember from a performance test we once did is, that the results
are a lot more realistic, better and more stable, if you try to decouple the startup
of
the different clients a little bit, so they are not all in the same section of code at
the same time.
We inserted random
On Sun, Feb 25, 2001 at 12:41:28AM -0500, Tom Lane wrote:
Attached are graphs from more thorough runs of pgbench with a commit
delay that occurs only when at least N other backends are running active
transactions. ...
It's not entirely clear what set of parameters is best, but it is
At 00:42 25/02/01 -0800, Nathan Myers wrote:
The only really bad performers were (0), (10k,1), (100k,20). The best
were (30k,1) and (30k,10), although (30k,5) also did well except at 40.
Why would 30k be a magic delay, regardless of siblings? What happened
at 40?
I had assumed that 40 was
-Original Message-
From: Tom Lane
Attached are graphs from more thorough runs of pgbench with a commit
delay that occurs only when at least N other backends are running active
transactions.
My initial try at this proved to be too noisy to tell much. The noise
seems to be
"Hiroshi Inoue" [EMAIL PROTECTED] writes:
How about the case with scaling factor 1 ? i.e Could your
proposal detect lock conflicts in reality ?
The code is set up to not count backends that are waiting on locks.
That is, to do a commit delay there must be at least N other backends
that are in
Philip Warner [EMAIL PROTECTED] writes:
At 00:42 25/02/01 -0800, Nathan Myers wrote:
The only really bad performers were (0), (10k,1), (100k,20). The best
were (30k,1) and (30k,10), although (30k,5) also did well except at 40.
Why would 30k be a magic delay, regardless of siblings? What
[EMAIL PROTECTED] (Nathan Myers) writes:
At low loads, it seems (100k,1) (brown +) did best by far, which seems
very odd. Even more odd, it did pretty well at very high loads but had
problems at intermediate loads.
In theory, all these variants should behave exactly the same for a
single
Tom Lane wrote:
Philip Warner [EMAIL PROTECTED] writes:
At 00:42 25/02/01 -0800, Nathan Myers wrote:
The only really bad performers were (0), (10k,1), (100k,20). The best
were (30k,1) and (30k,10), although (30k,5) also did well except at 40.
Why would 30k be a magic delay, regardless
Basically, I am not sure how much we lose by doing the delay after
returning COMMIT, and I know we gain quite a bit by enabling us to group
fsync calls.
If included, this should be an option only, and not the default option.
Sure it should never become the default, because the "D" in
On Sat, Feb 24, 2001 at 01:07:17AM -0500, Tom Lane wrote:
[EMAIL PROTECTED] (Nathan Myers) writes:
I see, I had it backwards: N=0 corresponds to "always delay", and
N=infinity (~0) is "never delay", or what you call zero delay. N=1 is
not interesting. N=M/2 or N=sqrt(M) or N=log(M)
Attached are graphs from more thorough runs of pgbench with a commit
delay that occurs only when at least N other backends are running active
transactions.
My initial try at this proved to be too noisy to tell much. The noise
seems to be coming from WAL checkpoints that occur during a run and
At 00:41 25/02/01 -0500, Tom Lane wrote:
Comments anyone?
Don't suppose you could post the original data?
Philip Warner| __---_
Albatross Consulting Pty. Ltd. |/ - \
(A.B.N. 75 008 659
Philip Warner [EMAIL PROTECTED] writes:
Don't suppose you could post the original data?
Sure.
regards, tom lane
commit_delay = 0
commit_siblings = 1
CHECKPOINT
transaction type: TPC-B (sort of)
scaling factor: 10
number of clients: 1
number of transactions per client:
Looking at the XLOG stuff, I notice that we already have a field
(logRec) in the per-backend PROC structures that shows whether a
transaction is currently in progress with at least one change made
(ie at least one XLOG entry written).
It would be very easy to extend the existing code so that the
Bruce Momjian [EMAIL PROTECTED] writes:
Why not just set a flag in there when someone nears commit and clear
when they are about to commit?
Define "nearing commit", in such a way that you can specify where you
plan to set that flag.
Is there significant time between entry of
Looking at the XLOG stuff, I notice that we already have a field
(logRec) in the per-backend PROC structures that shows whether a
transaction is currently in progress with at least one change made
(ie at least one XLOG entry written).
It would be very easy to extend the existing code so
Bruce Momjian [EMAIL PROTECTED] writes:
Why not just set a flag in there when someone nears commit and clear
when they are about to commit?
Define "nearing commit", in such a way that you can specify where you
plan to set that flag.
regards, tom lane
Bruce Momjian [EMAIL PROTECTED] writes:
Is there significant time between entry of CommitTransaction() and the
fsync()? Maybe not.
I doubt it. No I/O anymore, anyway, unless the commit record happens to
overrun an xlog block boundary.
That's what I was afraid of. Since we don't write
On Fri, Feb 23, 2001 at 11:32:21AM -0500, Tom Lane wrote:
A further refinement, still quite cheap to implement since the info is
in the PROC struct, would be to not count backends that are blocked
waiting for locks. These guys are less likely to be ready to commit
in the next few
On Fri, Feb 23, 2001 at 11:32:21AM -0500, Tom Lane wrote:
A further refinement, still quite cheap to implement since the info is
in the PROC struct, would be to not count backends that are blocked
waiting for locks. These guys are less likely to be ready to commit
in the next few
[EMAIL PROTECTED] (Nathan Myers) writes:
Comments? What should the threshold N be ... or do we need to make
that a tunable parameter?
Once you make it tuneable, you're stuck with it. You can always add
a knob later, after somebody discovers a real need.
If we had a good idea what the
[EMAIL PROTECTED] (Nathan Myers) writes:
Comments? What should the threshold N be ... or do we need to make
that a tunable parameter?
Once you make it tuneable, you're stuck with it. You can always add
a knob later, after somebody discovers a real need.
If we had a good idea what
And if they have, should we be more
likely to fsync() in the future?
I meant more likely to sleep().
You mean less likely. My thought for a self-adjusting delay was to
ratchet the delay up a little every time it succeeds in avoiding an
fsync, and down a little every time it fails to do
Bruce Momjian [EMAIL PROTECTED] writes:
A self-adjusting delay might well be a great idea, BTW, but I'm trying
to be conservative about how much complexity we should add right now.
OH, so you are saying N backends should have dirtied buffers before
doing the delay? Hmm, that seems almost
Bruce Momjian [EMAIL PROTECTED] writes:
It could be tough. Imagine the delay increasing to 3 seconds? Seems
there has to be an upper bound on the sleep. The more you delay, the
more likely you will be to find someone to fsync you.
Good point, and an excellent illustration of the fact that
On Fri, Feb 23, 2001 at 05:18:19PM -0500, Tom Lane wrote:
[EMAIL PROTECTED] (Nathan Myers) writes:
Comments? What should the threshold N be ... or do we need to make
that a tunable parameter?
Once you make it tuneable, you're stuck with it. You can always add
a knob later, after
When thinking about tuning N, I like to consider what are the interesting
possible values for N:
0: Ignore any other potential committers.
1: The minimum possible responsiveness to other committers.
5: Tom's guess for what might be a good choice.
10: Harry's guess.
~0: Always
Bruce Momjian [EMAIL PROTECTED] writes:
It could be tough. Imagine the delay increasing to 3 seconds? Seems
there has to be an upper bound on the sleep. The more you delay, the
more likely you will be to find someone to fsync you.
Good point, and an excellent illustration of the fact
Bruce Momjian [EMAIL PROTECTED] writes:
Can we put the backends to sleep waiting for a lock, and have them wake
up later?
Locks don't have timeouts. There is no existing mechanism that will
serve this purpose; we'll have to create a new one.
regards, tom lane
On Fri, Feb 23, 2001 at 06:37:06PM -0500, Bruce Momjian wrote:
When thinking about tuning N, I like to consider what are the interesting
possible values for N:
0: Ignore any other potential committers.
1: The minimum possible responsiveness to other committers.
5: Tom's guess
It seems to me that comparing various choices (and other, more
interesting, algorithms) to the N=1 case would be more productive
than comparing them to the N=0 case, so releasing at N=1 would yield
better statistics for actually tuning in 7.2.
We don't release code because it has
Bruce Momjian [EMAIL PROTECTED] writes:
Can we put the backends to sleep waiting for a lock, and have them wake
up later?
Locks don't have timeouts. There is no existing mechanism that will
serve this purpose; we'll have to create a new one.
That is what I suspected.
Having thought
At 21:31 23/02/01 -0500, Bruce Momjian wrote:
Now, I know many will complain that we are returning commit while not
having the stuff on the platter.
You're definitely right there.
Maybe they do, but it seems
the benefit of grouped fsyncs() is large enough that many will say they
would rather
At 21:31 23/02/01 -0500, Bruce Momjian wrote:
Now, I know many will complain that we are returning commit while not
having the stuff on the platter.
You're definitely right there.
Maybe they do, but it seems
the benefit of grouped fsyncs() is large enough that many will say they
would
At 11:32 23/02/01 -0500, Tom Lane wrote:
Looking at the XLOG stuff, I notice that we already have a field
(logRec) in the per-backend PROC structures that shows whether a
transaction is currently in progress with at least one change made
(ie at least one XLOG entry written).
Would it be worth
At 23:14 23/02/01 -0500, Bruce Momjian wrote:
There is one more thing. Even though the kernel says the data is on the
platter, it still may not be there.
This is true, but it does not mean we should say 'the disk is slightly
unreliable, so we can be too'. Also, IIRC, the last time this was
On Fri, Feb 23, 2001 at 09:05:20PM -0500, Bruce Momjian wrote:
It seems to me that comparing various choices (and other, more
interesting, algorithms) to the N=1 case would be more productive
than comparing them to the N=0 case, so releasing at N=1 would yield
better statistics
At 23:14 23/02/01 -0500, Bruce Momjian wrote:
There is one more thing. Even though the kernel says the data is on the
platter, it still may not be there.
This is true, but it does not mean we should say 'the disk is slightly
unreliable, so we can be too'. Also, IIRC, the last time this
Bruce Momjian [EMAIL PROTECTED] writes:
My idea would be to let committing backends return "COMMIT" to the user,
and set a need_fsync flag that is guaranteed to cause an fsync within X
milliseconds. This way, if other backends commit in the next X
millisecond, they can all use one
Bruce Momjian [EMAIL PROTECTED] writes:
My idea would be to let committing backends return "COMMIT" to the user,
and set a need_fsync flag that is guaranteed to cause an fsync within X
milliseconds. This way, if other backends commit in the next X
millisecond, they can all use one fsync().
At 21:31 23/02/01 -0500, Bruce Momjian wrote:
Now, I know many will complain that we are returning commit while not
having the stuff on the platter.
You're definitely right there.
Maybe they do, but it seems
the benefit of grouped fsyncs() is large enough that many will say they
would
Philip Warner [EMAIL PROTECTED] writes:
It may have been much earler in the debate, but has anyone checked to see
what the maximum possible gains might be - or is it self-evident to people
who know the code?
fsync off provides an upper bound to the speed achievable from being
smarter about
Preliminary results from experimenting with an
N-transactions-must-be-running-to-cause-commit-delay heuristic are
attached. It seems to be a pretty definite win. I'm currently running
a more extensive set of cases on another machine for comparison.
The test case is pgbench, unmodified, but run
[EMAIL PROTECTED] (Nathan Myers) writes:
I see, I had it backwards: N=0 corresponds to "always delay", and
N=infinity (~0) is "never delay", or what you call zero delay. N=1 is
not interesting. N=M/2 or N=sqrt(M) or N=log(M) might be interesting,
where M is the number of backends, or the
Philip Warner [EMAIL PROTECTED] writes:
It may have been much earler in the debate, but has anyone checked to see
what the maximum possible gains might be - or is it self-evident to people
who know the code?
fsync off provides an upper bound to the speed achievable from being
smarter
47 matches
Mail list logo
