Argh. It's just because I used -P5. It's a bit confusing that the other
options are per second, and this is per interval...
I agree, but I'm unsure of a fix, beyond what is already done which is to
show units next to the figures...
ISTM that people expect "tps" for performance, even on severa
On 2015/10/22 18:20, Fabien COELHO wrote:
>>
>>> progress: 365.0 s, 18392.1 tps, lat 94.857 ms stddev 23.917, lag 90.980
>>> ms, 106244 skipped
>>>
>>> 100k skipped transactions at a rate limit of 40k? That doesn't seem right.
>>
>> Argh. It's just because I used -P5. It's a bit confusing that the
Hello Andres,
pgbench postgres -M prepared -c 72 -j 72 -P 5 -T 3600 -R4 -L100
progress: 365.0 s, 18392.1 tps, lat 94.857 ms stddev 23.917, lag 90.980 ms,
106244 skipped
100k skipped transactions at a rate limit of 40k? That doesn't seem right.
Argh. It's just because I used -P5. It's
On 2015-10-20 20:55:46 +0200, Andres Freund wrote:
> On 2014-08-14 15:01:53 +0200, Fabien COELHO wrote:
> >
> > Add --limit to limit latency under throttling
> >
> > Under throttling, transactions are scheduled for execution at certain times.
> > Transactions may be far behind schedule and the sy
On 2014-08-14 15:01:53 +0200, Fabien COELHO wrote:
>
> Add --limit to limit latency under throttling
>
> Under throttling, transactions are scheduled for execution at certain times.
> Transactions may be far behind schedule and the system may catch up with the
> load later. This option allows to
On 10/13/14, 1:54 PM, Heikki Linnakangas wrote:
Greg Smith, I'd still appreciate it if you could take a look at this,
to check how this will work for pgbench-tools.
I'll do a QA pass on the committed version looking for issues, and
update the toolchain I publish to be compatible with it along
On 10/09/2014 10:39 PM, Fabien COELHO wrote:
One thing bothers me with the log format. Here's an example:
0 81 4621 0 1412881037 912698 3005
0 82 6173 0 1412881037 914578 4304
0 83 skipped 0 1412881037 914578 5217
0 83 skipped 0 1412881037 914578 5099
0 83 4722 0 1412881037 916203 310
One thing bothers me with the log format. Here's an example:
0 81 4621 0 1412881037 912698 3005
0 82 6173 0 1412881037 914578 4304
0 83 skipped 0 1412881037 914578 5217
0 83 skipped 0 1412881037 914578 5099
0 83 4722 0 1412881037 916203 3108
0 84 4142 0 1412881037 918023 2333
0 85 2465
On 10/05/2014 10:43 AM, Fabien COELHO wrote:
Hello Heikki,
Here are new patches, again the first one is just refactoring, and the second
one contains this feature. I'm planning to commit the first one shortly, and
the second one later after people have had a chance to look at it.
I looked at
Hello Heikki,
Here are new patches, again the first one is just refactoring, and the second
one contains this feature. I'm planning to commit the first one shortly, and
the second one later after people have had a chance to look at it.
I looked at it. It looks ok, but for a few spurious spac
On 09/15/2014 08:46 PM, Fabien COELHO wrote:
I'm not sure I like the idea of printing a percentage. It might be
unclear what the denominator was if somebody feels the urge to work
back to the actual number of skipped transactions. I mean, I guess
it's probably just the value you passed to -R,
I'm not sure I like the idea of printing a percentage. It might be
unclear what the denominator was if somebody feels the urge to work
back to the actual number of skipped transactions. I mean, I guess
it's probably just the value you passed to -R, so maybe that's easy
enough, but then why bot
On Mon, Sep 15, 2014 at 6:34 AM, Heikki Linnakangas
wrote:
> Please have a look. I have not looked at the docs changes yet.
>
> One thing that needs some thinking and changing is the progress reporting.
> It currently looks like this:
>
> progress: 1.0 s, 4863.0 tps, lat 3.491 ms stddev 2.487, lag
On 09/13/2014 11:25 AM, Fabien COELHO wrote:
[about logging...]
Here is an attempt at updating the log features, including the aggregate
and sampling stuff, with skipped transactions under throttling.
I moved the logging stuff into a function which is called when a
transaction is skipped or
On Sat, Sep 13, 2014 at 4:25 AM, Fabien COELHO wrote:
>> [about logging...]
>
> Here is an attempt at updating the log features, including the aggregate and
> sampling stuff, with skipped transactions under throttling.
>
> I moved the logging stuff into a function which is called when a transactio
[about logging...]
Here is an attempt at updating the log features, including the aggregate
and sampling stuff, with skipped transactions under throttling.
I moved the logging stuff into a function which is called when a
transaction is skipped or finished.
From a log file format perspect
On 09/12/2014 08:59 PM, Fabien COELHO wrote:
The output would look something like this (modified from the manual's example
by hand, so the numbers don't add up):
0 199 2241 0 1175850568 995598 1020
0 200 2465 0 1175850568 998079 1010
0 201 skipped 1175850569 608 3011
0 202 skipped 1175850569 6
On Fri, Sep 12, 2014 at 2:27 PM, Gregory Smith wrote:
> If there's a good case that the whole format needs to be changed anyway,
> like adding a new field, then we might as well switch to fractional epoch
> timestamps too now though. When I added timestamps to the latency log in
> 8.3, parsers th
On 9/10/14, 10:57 AM, Fabien COELHO wrote:
Indeed. I think that people do not like it to change. I remember that
I suggested to change timestamps to ".yy" instead of the
unreadable " yyy", and be told not to, because some people have
tool which process the output so the format MUST
The output would look something like this (modified from the manual's example
by hand, so the numbers don't add up):
0 199 2241 0 1175850568 995598 1020
0 200 2465 0 1175850568 998079 1010
0 201 skipped 1175850569 608 3011
0 202 skipped 1175850569 608 2400
0 203 skipped 1175850569 608 1000
0 2
(1) ISTM that the + 0.5 which remains in the PoissonRand computation comes
from the previous integer approach and is not needed here. If I'm not
mistaken the formula should be plain:
-log(uniform) * center
No. The +0.5 is to round the result to the nearest integer, instead of
truncati
On 09/11/2014 05:16 PM, Fabien COELHO wrote:
How should skipped transactions should be taken into account in the log file
output, with and without aggregation? I assume we'll want to have some trace
of skipped transactions in the logs.
The problem with this point is that how to report somethi
On 09/11/2014 03:36 PM, Fabien COELHO wrote:
Hello Heikki,
Now that I've finished the detour and committed and backpatched the changes
to the way latency is calculated, we can get back to this patch. It needs to
be rebased.
Before rebasing, I think that there are a few small problems with th
How should skipped transactions should be taken into account in the log file
output, with and without aggregation? I assume we'll want to have some trace
of skipped transactions in the logs.
The problem with this point is that how to report something "not done" is
unclear, especially as the
Hello Heikki
Now that I've finished the detour and committed and backpatched the changes
to the way latency is calculated, we can get back to this patch. It needs to
be rebased.
Here is the rebase, which seems ok.
See also the small issues raised aboud getPoissonRand in another email.
--
F
(3) I wish that the maximum implied multiplier could be explicitely
documented in the source code. From pg_rand48 source code, I think
that it is 33.27106466687737
Small possibly buggy code attached, to show how I computed the above
figure.
--
Fabien.#include
#include
int main(void
Hello Heikki,
Now that I've finished the detour and committed and backpatched the changes
to the way latency is calculated, we can get back to this patch. It needs to
be rebased.
Before rebasing, I think that there are a few small problems with the
modification applied to switch from an int
On 08/30/2014 07:16 PM, Fabien COELHO wrote:
+ if (latency_limit)
+ printf("number of transactions above the %.1f ms latency limit: "
INT64_FORMAT "\n",
+ latency_limit / 1000.0, latency_late);
+
Any reason not to report a percentage here?
Yes:
On 09/10/2014 05:47 PM, Mitsumasa KONDO wrote:
Hi,
I find typo in your patch. Please confirm.
@line 239
- agg->sum2_lag = 0;
+ agg->sum_lag = 0;
Ah thanks, cood catch!
And back patch is welcome for me.
I've committed and backpatched this, as well as a patch to refactor the
way the Poiss
On 09/10/2014 11:28 AM, Heikki Linnakangas wrote:
On 09/10/2014 05:57 PM, Fabien COELHO wrote:
Hello Heikki,
I looked closer at the this, and per Jan's comments, realized that we don't
log the lag time in the per-transaction log file. I think that's a serious
omission; when --rate is used, th
On 09/10/2014 05:57 PM, Fabien COELHO wrote:
Hello Heikki,
I looked closer at the this, and per Jan's comments, realized that we don't
log the lag time in the per-transaction log file. I think that's a serious
omission; when --rate is used, the schedule lag time is important information
to mak
Hello Heikki,
I looked closer at the this, and per Jan's comments, realized that we don't
log the lag time in the per-transaction log file. I think that's a serious
omission; when --rate is used, the schedule lag time is important information
to make sense of the result. I think we have to ap
Hi,
I find typo in your patch. Please confirm.
@line 239
- agg->sum2_lag = 0;
+ agg->sum_lag = 0;
And back patch is welcome for me.
Best Regards,
--
Mitsumasa KONDO
On 09/09/2014 03:35 PM, Fabien COELHO wrote:
Hello Heikki,
I think we have to reconsider what we're reporting in 9.4, when --rate
is enabled, even though it's already very late in the release cycle.
It's a bad idea to change the definition of latency between 9.4 and 9.5,
so let's get it right
Hello Heikki,
I think we have to reconsider what we're reporting in 9.4, when --rate
is enabled, even though it's already very late in the release cycle.
It's a bad idea to change the definition of latency between 9.4 and 9.5,
so let's get it right in 9.4.
Indeed.
As per the attached patch.
On 09/09/2014 01:49 PM, Heikki Linnakangas wrote:
I think we have to reconsider what we're reporting in 9.4, when --rate
is enabled, even though it's already very late in the release cycle.
It's a bad idea to change the definition of latency between 9.4 and 9.5,
so let's get it right in 9.4.
As
On 09/05/2014 06:38 PM, Jan Wieck wrote:
On 09/05/2014 10:12 AM, Fabien COELHO wrote:
Note that despite pg appaling latency performance, in may stay well over
the 90% limit, or even 100%: when things are going well a lot of
transaction run in about ms, while when things are going bad transaction
On 09/05/2014 10:12 AM, Fabien COELHO wrote:
Note that despite pg appaling latency performance, in may stay well over
the 90% limit, or even 100%: when things are going well a lot of
transaction run in about ms, while when things are going bad transactions
would take a long time (although possibl
That model might make some sense if you think e.g. of a web application,
where the web server has a timeout for how long it waits to get a
database connection from a pool, but once a query is started, the
transaction is considered a succeess no matter how long it takes. The
latency limit would b
On 08/27/2014 04:08 AM, Heikki Linnakangas wrote:
That model might make some sense if you think e.g. of a web application,
where the web server has a timeout for how long it waits to get a
database connection from a pool, but once a query is started, the
transaction is considered a succeess no ma
+ if (latency_limit)
+ printf("number of transactions above the %.1f ms latency limit: "
INT64_FORMAT "\n",
+ latency_limit / 1000.0, latency_late);
+
Any reason not to report a percentage here?
Yes: I did not thought of it.
Here is a v7, with a
Hi,
I generally want to say that having a feature like this feels *very*
helpful to me. Lots of pg development hasn't really paid attention to
anything but the final pgbench results...
On 2014-08-29 19:48:43 +0200, Fabien COELHO wrote:
> + if (latency_limit)
> + printf("number of
Hello Heikki,
[...] I would be fine with both.
After giving it some thought, ISTM better to choose consistency over
intuition, and have latency under throttling always defined wrt the
scheduled start time and not the actual start time, even if having a
latency of 1 ms for an OLTP load
Hello Heikki,
This now begs the question:
In --rate mode, shouldn't the reported transaction latency also be calculated
from the *scheduled* start time, not the time the transaction actually
started? Otherwise we're using two different definitions of "latency", one
for the purpose of the li
On 08/27/2014 08:05 PM, Fabien COELHO wrote:
[...]
Yeah, something like that. I don't think it would be necessary to set
statement_timeout, you can inject that in your script or postgresql.conf if
you want. I don't think aborting a transaction that's already started is
necessary either. You c
[...]
Yeah, something like that. I don't think it would be necessary to set
statement_timeout, you can inject that in your script or postgresql.conf if
you want. I don't think aborting a transaction that's already started is
necessary either. You could count it as LATE, but let it finish fi
On 08/27/2014 06:08 PM, Fabien COELHO wrote:
I've submitted this "simple" lag limit version because being able to
measure quickly and simply (un)responsiveness seems like a good idea,
especially given the current state of things.
Ok, fair enough. I don't think doing a "latency limit" would be s
As for an actual "latency limit" under throttling, this is significantly
more tricky and invasive to implement... ISTM that it would mean:
[...]
Yeah, something like that. I don't think it would be necessary to set
statement_timeout, you can inject that in your script or postgresql.conf if
On 08/27/2014 02:37 PM, Fabien COELHO wrote:
As for an actual "latency limit" under throttling, this is significantly
more tricky and invasive to implement... ISTM that it would mean:
- if the tx is not stated an the latency is already consummed, SKIP++.
- if the tx is after its schedule
Hello Heikki,
[...]
With a latency limit on when the query should finish, as opposed to how
late it can start, it's a lot easier to give a number. For example, your
requirements might state that a user must always get a response to a click on
a web page in 200 ms, so you set the limit to 200
On 08/27/2014 12:41 PM, Fabien COELHO wrote:
Hello Heikki,
I find the definition of the latency limit a bit strange. It's a limit on how
late a transaction can *start* compared to it's scheduled starting time, not
how long a query is allowed to last.
Yes. This is what can be done easily with
Hello Heikki,
I find the definition of the latency limit a bit strange. It's a limit on how
late a transaction can *start* compared to it's scheduled starting time, not
how long a query is allowed to last.
Yes. This is what can be done easily with pgbench under throttling. Note
that if tran
On 08/27/2014 03:47 AM, Rukh Meski wrote:
Hi Fabien,
On Tue, Aug 26, 2014 at 04:07 AM, Fabien COELHO wrote:
Please find attached a new version which fixes these two points.
Indeed it does. Marking the patch ready for a committer.
I find the definition of the latency limit a bit strange.
Hi Fabien,
On Tue, Aug 26, 2014 at 04:07 AM, Fabien COELHO wrote:
>
> Please find attached a new version which fixes these two points.
Indeed it does. Marking the patch ready for a committer.
Thanks,
♜
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes t
Marking Waiting for Author until these small issues have been fixed.
I've put it back to "Needs review". Feel free to set it to "Ready" if it
is ok for you.
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.po
Hello Rukh,
I have reviewed this patch.
Thanks!
[...] I get: pgbench: invalid option -- L
Which appears to be caused by the fact that the call to getopt_long()
has not been updated to reflect the new parameter.
Indeed, I only tested/used it with the --limit= syntax.
Also this part:
+
Hi Fabien,
On Sun, Aug 24, 2014 at 9:16 AM, Fabien COELHO wrote:
> Find attached a new version:
> - fix dropped percent computation in the final report
> - simplify progress report code
I have reviewed this patch.
Is the patch in a patch format which has context?
Yes.
Does it apply cleanly
Add --limit to limit latency under throttling
Under throttling, transactions are scheduled for execution at certain times.
Transactions may be far behind schedule and the system may catch up with the
load later. This option allows to change this behavior by skipping
transactions which are to
Add --limit to limit latency under throttling
Under throttling, transactions are scheduled for execution at certain
times. Transactions may be far behind schedule and the system may catch up
with the load later. This option allows to change this behavior by
skipping transactions which are too
59 matches
Mail list logo