Re: [HACKERS] pgbench --latency-limit option

[Fabien COELHO]

Hello Robert & Tatsuo,

Some paraphrasing and additional comments.


$ pgbench -p 11002 --rate 2 --latency-limit 1 -c 10 -T 10 test


You are targetting 2 tps over 10 connections, so that is about one 
transaction every 5 seconds for each connection, the target is about 20 
transactions in 10 seconds. You want transaction latency below 1 *ms*.



number of transactions actually processed: 16


The stochastic process scheduled 16 transactions during the 10 seconds, 
1.6 tps. In the long run it should be close to 2 tps, if the stochastic 
process does its job as expected.



number of transactions skipped: 0 (0.000 %)


All transactions were started (i.e. no transaction was skipped).


number of transactions above the 1.0 ms latency limit: 16 (100.000 %)


But none responded within 1 ms, they were all late.


latency average: 5.518 ms
latency stddev: 1.834 ms


Indeed, unlikely to be under 1 ms.


In my understanding, this says: any transaction takes longer than the
duration specified by --latency-limit (in this case 1.0 ms) will not
be sent the sever.


We cannot know that a transaction will take longer before running it.


I don't think that's what it says.  There seem to be three points here:

1. If the transaction is sent to the server, we'll check whether it
runs for longer than the amount of time specified by the limit; if so,
it will be reported separately.  This is true with or without --rate.


Yes.


2. If --rate is used, we'll calculate the latency for each statement
based on the time it was due to be sent, rather than the time it
actually got sent.  (This is further clarified in the documentation
for --rate.)


Yes. The idea is that the client wanted (say a web server) to send a 
transaction a time t, but due to the load or whatever it may not have been 
able to send it at that time, but it sends it later.


3. If --rate is used and the server is so far behind that 
--latency-limit cannot possibly be met, then we'll skip sending the 
query at all.


Yes. The time when the client finally get to send the transaction, the 
current time is beyond schedule + delay limit, no way to get an answer
in time, this simulates a client timeout, where the client gives up 
getting an answer.



In your example, you've got 10 connections and are trying to run at 2
tps, so to avoid having to start skipping things you need transaction
response times to be <~ 5 ms.  The actual response time is ~5.5ms, so
if you ran the test for longer I think you would see some skips.


Probably no skips though, because the response time needed is below 5 
*seconds*, not ms : 2 tps on 10 connections, 1 transaction every 5 seconds 
for each connection.


--
Fabien.


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pgbench --latency-limit option

[Robert Haas]

On Wed, Dec 23, 2015 at 11:23 AM, Fabien COELHO  wrote:
>>> Probably no skips though, because the response time needed is below 5
>>> *seconds*, not ms : 2 tps on 10 connections, 1 transaction every 5
>>> seconds
>>> for each connection.
>>
>> Oops.  Right.  But why did this test only run 16 transactions in total
>> instead of 20?
>
> Because the schedule is based on a stochastic process, transactions are not
> set regularly (that would induce patterns and is not representative of
> real-life load) but randomly.
>
> The long term average is expected to converge to 2 tps, but on a short run
> it may differ significantly.

Hmm.  Is that documented somewhere?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pgbench --latency-limit option

[Robert Haas]

On Tue, Dec 22, 2015 at 9:28 PM, Tatsuo Ishii  wrote:
> While playing with 9.5's pgbench, I faced with a strange behavior.
>
> $ pgbench -p 11002 --rate 2 --latency-limit 1 -c 10 -T 10 test
> starting vacuum...end.
> transaction type: TPC-B (sort of)
> scaling factor: 1
> query mode: simple
> number of clients: 10
> number of threads: 1
> duration: 10 s
> number of transactions actually processed: 16
> number of transactions skipped: 0 (0.000 %)
> number of transactions above the 1.0 ms latency limit: 16 (100.000 %)
> latency average: 5.518 ms
> latency stddev: 1.834 ms
> rate limit schedule lag: avg 0.694 (max 1.823) ms
> tps = 1.599917 (including connections establishing)
> tps = 1.600319 (excluding connections establishing)
>
> From the pgbench manual:
>
>   --latency-limit=limit
>   
>
> Transaction which last more than limit milliseconds
> are counted and reported separately, as late.
>
>
> When throttling is used (--rate=...), transactions that
> lag behind schedule by more than limit ms, and thus
> have no hope of meeting the latency limit, are not sent to the server
> at all. They are counted and reported separately as
> skipped.
>
> In my understanding, this says: any transaction takes longer than the
> duration specified by --latency-limit (in this case 1.0 ms) will not
> be sent the sever.

I don't think that's what it says.  There seem to be three points here:

1. If the transaction is sent to the server, we'll check whether it
runs for longer than the amount of time specified by the limit; if so,
it will be reported separately.  This is true with or without --rate.

2. If --rate is used, we'll calculate the latency for each statement
based on the time it was due to be sent, rather than the time it
actually got sent.  (This is further clarified in the documentation
for --rate.)

3. If --rate is used and the server is so far behind that
--latency-limit cannot possibly be met, then we'll skip sending the
query at all.

In your example, you've got 10 connections and are trying to run at 2
tps, so to avoid having to start skipping things you need transaction
response times to be <~ 5 ms.  The actual response time is ~5.5ms, so
if you ran the test for longer I think you would see some skips.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pgbench --latency-limit option

[Robert Haas]

On Wed, Dec 23, 2015 at 9:52 AM, Fabien COELHO  wrote:
>> In your example, you've got 10 connections and are trying to run at 2
>> tps, so to avoid having to start skipping things you need transaction
>> response times to be <~ 5 ms.  The actual response time is ~5.5ms, so
>> if you ran the test for longer I think you would see some skips.
>
> Probably no skips though, because the response time needed is below 5
> *seconds*, not ms : 2 tps on 10 connections, 1 transaction every 5 seconds
> for each connection.

Oops.  Right.  But why did this test only run 16 transactions in total
instead of 20?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pgbench --latency-limit option

[Fabien COELHO]

Probably no skips though, because the response time needed is below 5
*seconds*, not ms : 2 tps on 10 connections, 1 transaction every 5 seconds
for each connection.


Oops.  Right.  But why did this test only run 16 transactions in total
instead of 20?


Because the schedule is based on a stochastic process, transactions are 
not set regularly (that would induce patterns and is not representative of 
real-life load) but randomly.


The long term average is expected to converge to 2 tps, but on a short run 
it may differ significantly.


--
Fabien.


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pgbench --latency-limit option

[Fabien COELHO]

[...]
Because the schedule is based on a stochastic process, transactions are not
set regularly (that would induce patterns and is not representative of
real-life load) but randomly.

The long term average is expected to converge to 2 tps, but on a short run
it may differ significantly.


Hmm.  Is that documented somewhere?


Sure, see --rate option in pgbench doc, which states:

"The rate is targeted by starting transactions along a Poisson-distributed 
schedule time line."


The impact on the tps is implied, though.

--
Fabien.


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] pgbench --latency-limit option

[Tatsuo Ishii]

While playing with 9.5's pgbench, I faced with a strange behavior.

$ pgbench -p 11002 --rate 2 --latency-limit 1 -c 10 -T 10 test
starting vacuum...end.
transaction type: TPC-B (sort of)
scaling factor: 1
query mode: simple
number of clients: 10
number of threads: 1
duration: 10 s
number of transactions actually processed: 16
number of transactions skipped: 0 (0.000 %)
number of transactions above the 1.0 ms latency limit: 16 (100.000 %)
latency average: 5.518 ms
latency stddev: 1.834 ms
rate limit schedule lag: avg 0.694 (max 1.823) ms
tps = 1.599917 (including connections establishing)
tps = 1.600319 (excluding connections establishing)

>From the pgbench manual:

  --latency-limit=limit
  
   
Transaction which last more than limit milliseconds
are counted and reported separately, as late.
   
   
When throttling is used (--rate=...), transactions that
lag behind schedule by more than limit ms, and thus
have no hope of meeting the latency limit, are not sent to the server
at all. They are counted and reported separately as
skipped.

In my understanding, this says: any transaction takes longer than the
duration specified by --latency-limit (in this case 1.0 ms) will not
be sent the sever. In the case above all (16) transactions were behind
the latency limit 1.0 ms:

number of transactions above the 1.0 ms latency limit: 16 (100.000 %)

So the number of skipped transactions should be 16 (100%), rather
than:

number of transactions skipped: 0 (0.000 %)

in this case I think.

Am I missing something?

Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers