Hello Anders,
This v9 : - add "-b list" to show the list of builtins - remove the explicit --per-scripts-stats option, which is instead automatically set when several scripts are run or with per-command latencies (-r) - count scripts from 1 instead of 0 in the output I've left out: - removing -N/-S upward compatibility shorthands but I will not cry if they are removed - requiring percents instead of integer weights, because it is too constrained - your "array" remark as I did not understood itThanks to the restructuring and sharing of stats code, the patch does not change the loc count, although a few features are added.
-- Fabien.
diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml index 2517a3a..3edd65a 100644 --- a/doc/src/sgml/ref/pgbench.sgml +++ b/doc/src/sgml/ref/pgbench.sgml @@ -261,6 +261,25 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</> benchmarking arguments: <variablelist> + <varlistentry> + <term><option>-b</> <replaceable>scriptname[@weight]</></term> + <term><option>--builtin</> <replaceable>scriptname[@weight]</></term> + <listitem> + <para> + Add the specified builtin script to the list of executed scripts. + An optional integer weight after <literal>@</> allows to adjust the + probability of drawing the test. + Available builtin scripts are: <literal>tpcb-like</>, + <literal>simple-update</> and <literal>select-only</>. + The provided <repleacable>scriptname</> needs only to be a prefix + of the builtin name, hence <literal>simp</> would be enough to select + <literal>simple-update</>. + With special name <literal>list</>, show the list of builtin scripts + and exit immediately. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><option>-c</option> <replaceable>clients</></term> @@ -307,14 +326,15 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</> </varlistentry> <varlistentry> - <term><option>-f</option> <replaceable>filename</></term> - <term><option>--file=</option><replaceable>filename</></term> + <term><option>-f</> <replaceable>filename[@weight]</></term> + <term><option>--file=</><replaceable>filename[@weight]</></term> <listitem> <para> - Read transaction script from <replaceable>filename</>. + Add a transaction script read from <replaceable>filename</> to + the list of executed scripts. + An optional integer weight after <literal>@</> allows to adjust the + probability of drawing the test. See below for details. - <option>-N</option>, <option>-S</option>, and <option>-f</option> - are mutually exclusive. </para> </listitem> </varlistentry> @@ -404,10 +424,7 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</> <term><option>--skip-some-updates</option></term> <listitem> <para> - Do not update <structname>pgbench_tellers</> and - <structname>pgbench_branches</>. - This will avoid update contention on these tables, but - it makes the test case even less like TPC-B. + Shorthand for <option>-b simple-update@1</>. </para> </listitem> </varlistentry> @@ -499,9 +516,9 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</> Report the specified scale factor in <application>pgbench</>'s output. With the built-in tests, this is not necessary; the correct scale factor will be detected by counting the number of - rows in the <structname>pgbench_branches</> table. However, when testing - custom benchmarks (<option>-f</> option), the scale factor - will be reported as 1 unless this option is used. + rows in the <structname>pgbench_branches</> table. + However, when testing only custom benchmarks (<option>-f</> option), + the scale factor will be reported as 1 unless this option is used. </para> </listitem> </varlistentry> @@ -511,7 +528,7 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</> <term><option>--select-only</option></term> <listitem> <para> - Perform select-only transactions instead of TPC-B-like test. + Shorthand for <option>-b select-only@1</>. </para> </listitem> </varlistentry> @@ -661,7 +678,20 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</> <title>What is the <quote>Transaction</> Actually Performed in pgbench?</title> <para> - The default transaction script issues seven commands per transaction: + Pgbench executes test scripts chosen randomly from a specified list. + They include built-in scripts with <option>-b</> and + user-provided custom scripts with <option>-f</>. + Each script may be given a relative weight specified after a + <literal>@</> so as to change its drawing probability. + The default weight is <literal>1</>. + </para> + + <para> + The default builtin transaction script (also invoked with <option>-b tpcb-like</>) + issues seven commands per transaction over randomly chosen <literal>aid</>, + <literal>tid</>, <literal>bid</> and <literal>balance</>. + The scenario is inspired by the TPC-B benchmark, but is not actually TPC-B, + hence the name. </para> <orderedlist> @@ -675,9 +705,15 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</> </orderedlist> <para> - If you specify <option>-N</>, steps 4 and 5 aren't included in the - transaction. If you specify <option>-S</>, only the <command>SELECT</> is - issued. + If you select the <literal>simple-update</> builtin (also <option>-N</>), + steps 4 and 5 aren't included in the transaction. + This will avoid update contention on these tables, but + it makes the test case even less like TPC-B. + </para> + + <para> + If you select the <literal>select-only</> builtin (also <option>-S</>), + only the <command>SELECT</> is issued. </para> </refsect2> @@ -689,10 +725,7 @@ pgbench <optional> <replaceable>options</> </optional> <replaceable>dbname</> benchmark scenarios by replacing the default transaction script (described above) with a transaction script read from a file (<option>-f</option> option). In this case a <quote>transaction</> - counts as one execution of a script file. You can even specify - multiple scripts (multiple <option>-f</option> options), in which - case a random one of the scripts is chosen each time a client session - starts a new transaction. + counts as one execution of a script file. </para> <para> diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c index 30e8d2a..8540ac7 100644 --- a/src/bin/pgbench/pgbench.c +++ b/src/bin/pgbench/pgbench.c @@ -164,11 +164,10 @@ bool use_log; /* log transaction latencies to a file */ bool use_quiet; /* quiet logging onto stderr */ int agg_interval; /* log aggregates instead of individual * transactions */ +bool per_script_stats = false; /* whether to collect stats per script */ int progress = 0; /* thread progress report every this seconds */ -int progress_nclients = 0; /* number of clients for progress - * report */ -int progress_nthreads = 0; /* number of threads for progress - * report */ +int nclients = 1; /* number of clients */ +int nthreads = 1; /* number of threads */ bool is_connect; /* establish connection for each transaction */ bool is_latencies; /* report per-command latencies */ int main_pid; /* main process id used in log filename */ @@ -179,6 +178,8 @@ char *login = NULL; char *dbName; const char *progname; +#define WSEP '@' /* weight separator */ + volatile bool timer_exceeded = false; /* flag from signal handler */ /* variable definitions */ @@ -188,13 +189,39 @@ typedef struct char *value; /* its value */ } Variable; -#define MAX_FILES 128 /* max number of SQL script files allowed */ +#define MAX_SCRIPTS 128 /* max number of SQL scripts allowed */ #define SHELL_COMMAND_SIZE 256 /* maximum size allowed for shell command */ /* + * simple data structure to keep stats about something. + * probably the first value should be kept and used as an offset for + * better numerical stability... + */ +typedef struct +{ + int64_t count; /* how many values where encountered */ + double min; /* the minimum seen */ + double max; /* the maximum seen */ + double sum; /* sum of values */ + double sum2; /* sum of squared values */ +} SimpleStats; + +/* data structure to hold various statistics. + * it is used for interval statistics as well as file statistics. + */ +typedef struct +{ + long start_time; /* when the interval starts, for aggregates */ + int64 cnt; /* number of transactions */ + int64 skipped; /* number of transactions skipped under --rate + * and --latency-limit */ + SimpleStats latency; + SimpleStats lag; +} StatsData; + +/* * structures used in custom query mode */ - typedef struct { PGconn *con; /* connection handle to DB */ @@ -202,22 +229,20 @@ typedef struct int state; /* state No. */ int listen; /* 0 indicates that an async query has been * sent */ - int sleeping; /* 1 indicates that the client is napping */ + bool is_throttled; /* whether transaction throttling is done */ + bool sleeping; /* whether the client is napping */ bool throttling; /* whether nap is for throttling */ Variable *variables; /* array of variable definitions */ int nvariables; int64 txn_scheduled; /* scheduled start time of transaction (usec) */ instr_time txn_begin; /* used for measuring schedule lag times */ instr_time stmt_begin; /* used for measuring statement latencies */ - bool is_throttled; /* whether transaction throttling is done */ - int use_file; /* index in sql_files for this client */ - bool prepared[MAX_FILES]; + int use_file; /* index in sql_scripts for this client */ + bool prepared[MAX_SCRIPTS]; /* whether client prepared the script */ /* per client collected stats */ - int cnt; /* xacts count */ + int64 cnt; /* transaction count */ int ecnt; /* error count */ - int64 txn_latencies; /* cumulated latencies */ - int64 txn_sqlats; /* cumulated square latencies */ } CState; /* @@ -229,19 +254,14 @@ typedef struct pthread_t thread; /* thread handle */ CState *state; /* array of CState */ int nstate; /* length of state[] */ - instr_time start_time; /* thread start time */ - instr_time *exec_elapsed; /* time spent executing cmds (per Command) */ - int *exec_count; /* number of cmd executions (per Command) */ - unsigned short random_state[3]; /* separate randomness for each thread */ - int64 throttle_trigger; /* previous/next throttling (us) */ + unsigned short random_state[3]; /* separate randomness for each thread */ + int64 throttle_trigger; /* previous/next throttling (us) */ /* per thread collected stats */ + instr_time start_time; /* thread start time */ instr_time conn_time; - int64 throttle_lag; /* total transaction lag behind throttling */ - int64 throttle_lag_max; /* max transaction lag */ - int64 throttle_latency_skipped; /* lagging transactions - * skipped */ - int64 latency_late; /* late transactions */ + StatsData stats; + int64 latency_late; /* executed but late transactions */ } TState; #define INVALID_THREAD ((pthread_t) 0) @@ -273,80 +293,97 @@ typedef struct char *argv[MAX_ARGS]; /* command word list */ int cols[MAX_ARGS]; /* corresponding column starting from 1 */ PgBenchExpr *expr; /* parsed expression */ + SimpleStats stats; /* time spent in this command */ } Command; +/* SQL script to be executed, either a file or an internal script + */ typedef struct { - - long start_time; /* when does the interval start */ - int cnt; /* number of transactions */ - int skipped; /* number of transactions skipped under --rate - * and --latency-limit */ - - double min_latency; /* min/max latencies */ - double max_latency; - double sum_latency; /* sum(latency), sum(latency^2) - for - * estimates */ - double sum2_latency; - - double min_lag; - double max_lag; - double sum_lag; /* sum(lag) */ - double sum2_lag; /* sum(lag*lag) */ -} AggVals; - -static Command **sql_files[MAX_FILES]; /* SQL script files */ -static int num_files; /* number of script files */ + const char *name; + int weight; + Command **commands; + StatsData stats; +} SQLScript; + +static SQLScript sql_script[MAX_SCRIPTS]; +static int num_scripts; /* number of script in sql_script[] */ static int num_commands = 0; /* total number of Command structs */ +static int total_weight = 0; + static int debug = 0; /* debug flag */ -/* default scenario */ -static char *tpc_b = { - "\\set nbranches " CppAsString2(nbranches) " * :scale\n" - "\\set ntellers " CppAsString2(ntellers) " * :scale\n" - "\\set naccounts " CppAsString2(naccounts) " * :scale\n" - "\\setrandom aid 1 :naccounts\n" - "\\setrandom bid 1 :nbranches\n" - "\\setrandom tid 1 :ntellers\n" - "\\setrandom delta -5000 5000\n" - "BEGIN;\n" - "UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;\n" - "SELECT abalance FROM pgbench_accounts WHERE aid = :aid;\n" - "UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;\n" - "UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;\n" - "INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);\n" - "END;\n" -}; - -/* -N case */ -static char *simple_update = { - "\\set nbranches " CppAsString2(nbranches) " * :scale\n" - "\\set ntellers " CppAsString2(ntellers) " * :scale\n" - "\\set naccounts " CppAsString2(naccounts) " * :scale\n" - "\\setrandom aid 1 :naccounts\n" - "\\setrandom bid 1 :nbranches\n" - "\\setrandom tid 1 :ntellers\n" - "\\setrandom delta -5000 5000\n" - "BEGIN;\n" - "UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;\n" - "SELECT abalance FROM pgbench_accounts WHERE aid = :aid;\n" - "INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);\n" - "END;\n" -}; - -/* -S case */ -static char *select_only = { - "\\set naccounts " CppAsString2(naccounts) " * :scale\n" - "\\setrandom aid 1 :naccounts\n" - "SELECT abalance FROM pgbench_accounts WHERE aid = :aid;\n" -}; - /* Function prototypes */ static void setalarm(int seconds); static void *threadRun(void *arg); +static void doTxStats(TState*, CState*, instr_time*, bool, FILE*, StatsData*); -static void doLog(TState *thread, CState *st, FILE *logfile, instr_time *now, - AggVals *agg, bool skipped); +/* Define builtin test scripts */ +#define N_BUILTIN 3 +static struct { + char *name; /* very short name for -b ...*/ + char *desc; /* short description */ + char *script; /* actual pgbench script */ +} builtin_script[] = { +{ + "tpcb-like", + "<builtin: TPC-B (sort of)>", + "\\set nbranches " CppAsString2(nbranches) " * :scale\n" + "\\set ntellers " CppAsString2(ntellers) " * :scale\n" + "\\set naccounts " CppAsString2(naccounts) " * :scale\n" + "\\setrandom aid 1 :naccounts\n" + "\\setrandom bid 1 :nbranches\n" + "\\setrandom tid 1 :ntellers\n" + "\\setrandom delta -5000 5000\n" + "BEGIN;\n" + "UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;\n" + "SELECT abalance FROM pgbench_accounts WHERE aid = :aid;\n" + "UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;\n" + "UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;\n" + "INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);\n" + "END;\n" +}, +{ + "simple-update", + "<builtin: simple update>", + "\\set nbranches " CppAsString2(nbranches) " * :scale\n" + "\\set ntellers " CppAsString2(ntellers) " * :scale\n" + "\\set naccounts " CppAsString2(naccounts) " * :scale\n" + "\\setrandom aid 1 :naccounts\n" + "\\setrandom bid 1 :nbranches\n" + "\\setrandom tid 1 :ntellers\n" + "\\setrandom delta -5000 5000\n" + "BEGIN;\n" + "UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;\n" + "SELECT abalance FROM pgbench_accounts WHERE aid = :aid;\n" + "INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);\n" + "END;\n" +}, +{ + "select-only", + "<builtin: select only>", + "\\set naccounts " CppAsString2(naccounts) " * :scale\n" + "\\setrandom aid 1 :naccounts\n" + "SELECT abalance FROM pgbench_accounts WHERE aid = :aid;\n" +} }; + +static char * +find_builtin(const char *name, char **desc) +{ + int len = strlen(name), i; + + for (i = 0; i < N_BUILTIN; i++) + { + if (strncmp(builtin_script[i].name, name, len) == 0) + { + *desc = builtin_script[i].desc; + return builtin_script[i].script; + } + } + + fprintf(stderr, "no builtin found for \"%s\"\n", name); + exit(1); +} static void usage(void) @@ -366,11 +403,13 @@ usage(void) " --tablespace=TABLESPACE create tables in the specified tablespace\n" " --unlogged-tables create tables as unlogged tables\n" "\nBenchmarking options:\n" + " -b, --builtin=NAME@W add weighted buitin script among \"tpcb-like\"\n" + " \"simple-update\" and \"select-only\".\n" " -c, --client=NUM number of concurrent database clients (default: 1)\n" " -C, --connect establish new connection for each transaction\n" " -D, --define=VARNAME=VALUE\n" " define variable for use by custom script\n" - " -f, --file=FILENAME read transaction script from FILENAME\n" + " -f, --file=FILENAME@W add weighted transaction script from FILENAME\n" " -j, --jobs=NUM number of threads (default: 1)\n" " -l, --log write transaction times to log file\n" " -L, --latency-limit=NUM count transactions lasting more than NUM ms\n" @@ -378,12 +417,12 @@ usage(void) " -M, --protocol=simple|extended|prepared\n" " protocol for submitting queries (default: simple)\n" " -n, --no-vacuum do not run VACUUM before tests\n" - " -N, --skip-some-updates skip updates of pgbench_tellers and pgbench_branches\n" + " -N, --skip-some-updates same as \"-b simple-update@1\"\n" " -P, --progress=NUM show thread progress report every NUM seconds\n" " -r, --report-latencies report average latency per command\n" " -R, --rate=NUM target rate in transactions per second\n" " -s, --scale=NUM report this scale factor in output\n" - " -S, --select-only perform SELECT-only transactions\n" + " -S, --select-only same as \"-b select-only@1\"\n" " -t, --transactions=NUM number of transactions each client runs (default: 10)\n" " -T, --time=NUM duration of benchmark test in seconds\n" " -v, --vacuum-all vacuum all four standard tables before tests\n" @@ -580,6 +619,49 @@ getPoissonRand(TState *thread, int64 center) return (int64) (-log(uniform) * ((double) center) + 0.5); } +static void +initSimpleStats(SimpleStats * ss) +{ + memset(ss, 0, sizeof(SimpleStats)); +} + +static void +doSimpleStats(SimpleStats *ss, double val) +{ + if (ss->count == 0 || val < ss->min) + ss->min = val; + if (ss->count == 0 || val > ss->max) + ss->max = val; + ss->count ++; + ss->sum += val; + ss->sum2 += val * val; +} + +static void +appendSimpleStats(SimpleStats *acc, SimpleStats *ss) +{ + if (acc->count == 0 || ss->min < acc->min) + acc->min = ss->min; + if (acc->count == 0 || ss->max > acc->max) + acc->max = ss->max; + acc->count += ss->count; + acc->sum += ss->sum; + acc->sum2 += ss->sum2; +} + +static void +initStats(StatsData *sd, double start_time) +{ + sd->cnt = 0; + sd->skipped = 0; + initSimpleStats(& sd->latency); + initSimpleStats(& sd->lag); + + /* not necessarily overriden? */ + if (start_time) + sd->start_time = start_time; +} + /* call PQexec() and exit() on failure */ static void executeStatement(PGconn *con, const char *sql) @@ -1099,33 +1181,21 @@ clientDone(CState *st, bool ok) return false; /* always false */ } -static void -agg_vals_init(AggVals *aggs, instr_time start) +static int +chooseScript(TState *thread) { - /* basic counters */ - aggs->cnt = 0; /* number of transactions (includes skipped) */ - aggs->skipped = 0; /* xacts skipped under --rate --latency-limit */ + int i = 0, w = 0, wc = (int) getrand(thread, 0, total_weight - 1); - aggs->sum_latency = 0; /* SUM(latency) */ - aggs->sum2_latency = 0; /* SUM(latency*latency) */ + do { + w += sql_script[i++].weight; + } while (w <= wc); - /* min and max transaction duration */ - aggs->min_latency = 0; - aggs->max_latency = 0; - - /* schedule lag counters */ - aggs->sum_lag = 0; - aggs->sum2_lag = 0; - aggs->min_lag = 0; - aggs->max_lag = 0; - - /* start of the current interval */ - aggs->start_time = INSTR_TIME_GET_DOUBLE(start); + return i - 1; } /* return false iff client should be disconnected */ static bool -doCustom(TState *thread, CState *st, instr_time *conn_time, FILE *logfile, AggVals *agg) +doCustom(TState *thread, CState *st, instr_time *conn_time, FILE *logfile, StatsData *agg) { PGresult *res; Command **commands; @@ -1143,7 +1213,7 @@ doCustom(TState *thread, CState *st, instr_time *conn_time, FILE *logfile, AggVa top: INSTR_TIME_SET_ZERO(now); - commands = sql_files[st->use_file]; + commands = sql_script[st->use_file].commands; /* * Handle throttling once per transaction by sleeping. It is simpler to @@ -1179,18 +1249,15 @@ top: now_us = INSTR_TIME_GET_MICROSEC(now); while (thread->throttle_trigger < now_us - latency_limit) { - thread->throttle_latency_skipped++; - - if (logfile) - doLog(thread, st, logfile, &now, agg, true); - + doTxStats(thread, st, &now, true, logfile, agg); + /* next rendez-vous */ wait = getPoissonRand(thread, throttle_delay); thread->throttle_trigger += wait; st->txn_scheduled = thread->throttle_trigger; } } - st->sleeping = 1; + st->sleeping = true; st->throttling = true; st->is_throttled = true; if (debug) @@ -1200,27 +1267,13 @@ top: if (st->sleeping) { /* are we sleeping? */ - int64 now_us; - if (INSTR_TIME_IS_ZERO(now)) INSTR_TIME_SET_CURRENT(now); - now_us = INSTR_TIME_GET_MICROSEC(now); - if (st->txn_scheduled <= now_us) - { - st->sleeping = 0; /* Done sleeping, go ahead with next command */ - if (st->throttling) - { - /* Measure lag of throttled transaction relative to target */ - int64 lag = now_us - st->txn_scheduled; - - thread->throttle_lag += lag; - if (lag > thread->throttle_lag_max) - thread->throttle_lag_max = lag; - st->throttling = false; - } - } - else + if (INSTR_TIME_GET_MICROSEC(now) < st->txn_scheduled) return true; /* Still sleeping, nothing to do here */ + /* Else done sleeping, go ahead with next command */ + st->sleeping = false; + st->throttling = false; } if (st->listen) @@ -1244,47 +1297,24 @@ top: */ if (is_latencies) { - int cnum = commands[st->state]->command_num; - if (INSTR_TIME_IS_ZERO(now)) INSTR_TIME_SET_CURRENT(now); - INSTR_TIME_ACCUM_DIFF(thread->exec_elapsed[cnum], - now, st->stmt_begin); - thread->exec_count[cnum]++; + + /* although a mutex would make sense, the likelyhood of an issue + * is small and these are only stats which may be slightly false + */ + doSimpleStats(& commands[st->state]->stats, + INSTR_TIME_GET_DOUBLE(now) - + INSTR_TIME_GET_DOUBLE(st->stmt_begin)); } /* transaction finished: calculate latency and log the transaction */ if (commands[st->state + 1] == NULL) { - /* only calculate latency if an option is used that needs it */ - if (progress || throttle_delay || latency_limit) - { - int64 latency; - - if (INSTR_TIME_IS_ZERO(now)) - INSTR_TIME_SET_CURRENT(now); - - latency = INSTR_TIME_GET_MICROSEC(now) - st->txn_scheduled; - - st->txn_latencies += latency; - - /* - * XXX In a long benchmark run of high-latency transactions, - * this int64 addition eventually overflows. For example, 100 - * threads running 10s transactions will overflow it in 2.56 - * hours. With a more-typical OLTP workload of .1s - * transactions, overflow would take 256 hours. - */ - st->txn_sqlats += latency * latency; - - /* record over the limit transactions if needed. */ - if (latency_limit && latency > latency_limit) - thread->latency_late++; - } - - /* record the time it took in the log */ - if (logfile) - doLog(thread, st, logfile, &now, agg, false); + if (progress || throttle_delay || latency_limit || per_script_stats || logfile) + doTxStats(thread, st, &now, false, logfile, agg); + else + thread->stats.cnt ++; } if (commands[st->state]->type == SQL_COMMAND) @@ -1327,8 +1357,9 @@ top: if (commands[st->state] == NULL) { st->state = 0; - st->use_file = (int) getrand(thread, 0, num_files - 1); - commands = sql_files[st->use_file]; + st->use_file = num_scripts==1? 0: chooseScript(thread); + + commands = sql_script[st->use_file].commands; st->is_throttled = false; /* @@ -1371,7 +1402,7 @@ top: } /* Record transaction start time under logging, progress or throttling */ - if ((logfile || progress || throttle_delay || latency_limit) && st->state == 0) + if ((logfile || progress || throttle_delay || latency_limit || per_script_stats) && st->state == 0) { INSTR_TIME_SET_CURRENT(st->txn_begin); @@ -1659,7 +1690,7 @@ top: INSTR_TIME_SET_CURRENT(now); st->txn_scheduled = INSTR_TIME_GET_MICROSEC(now) + usec; - st->sleeping = 1; + st->sleeping = true; st->listen = 1; } @@ -1691,22 +1722,39 @@ top: else /* succeeded */ st->listen = 1; } + + /* after a meta command, immediately proceed with next command */ goto top; } return true; } +static void +doStats(StatsData *stats, bool skipped, double lat, double lag) +{ + stats->cnt ++; + + if (skipped) + /* no latency to record on skipped transactions */ + stats->skipped ++; + else + { + doSimpleStats(& stats->latency, lat); + + /* and possibly the same for schedule lag */ + if (throttle_delay) + doSimpleStats(& stats->lag, lag); + } +} + /* * print log entry after completing one transaction. */ static void -doLog(TState *thread, CState *st, FILE *logfile, instr_time *now, AggVals *agg, - bool skipped) +doLog(TState *thread, CState *st, FILE *logfile, instr_time *now, + StatsData *agg, bool skipped, double latency, double lag) { - double lag; - double latency; - /* * Skip the log entry if sampling is enabled and this row doesn't belong * to the random sample. @@ -1715,15 +1763,6 @@ doLog(TState *thread, CState *st, FILE *logfile, instr_time *now, AggVals *agg, pg_erand48(thread->random_state) > sample_rate) return; - if (INSTR_TIME_IS_ZERO(*now)) - INSTR_TIME_SET_CURRENT(*now); - - latency = (double) (INSTR_TIME_GET_MICROSEC(*now) - st->txn_scheduled); - if (skipped) - lag = latency; - else - lag = (double) (INSTR_TIME_GET_MICROSEC(st->txn_begin) - st->txn_scheduled); - /* should we aggregate the results or not? */ if (agg_interval > 0) { @@ -1733,39 +1772,7 @@ doLog(TState *thread, CState *st, FILE *logfile, instr_time *now, AggVals *agg, */ if (agg->start_time + agg_interval >= INSTR_TIME_GET_DOUBLE(*now)) { - agg->cnt += 1; - if (skipped) - { - /* - * there is no latency to record if the transaction was - * skipped - */ - agg->skipped += 1; - } - else - { - agg->sum_latency += latency; - agg->sum2_latency += latency * latency; - - /* first in this aggregation interval */ - if ((agg->cnt == 1) || (latency < agg->min_latency)) - agg->min_latency = latency; - - if ((agg->cnt == 1) || (latency > agg->max_latency)) - agg->max_latency = latency; - - /* and the same for schedule lag */ - if (throttle_delay) - { - agg->sum_lag += lag; - agg->sum2_lag += lag * lag; - - if ((agg->cnt == 1) || (lag < agg->min_lag)) - agg->min_lag = lag; - if ((agg->cnt == 1) || (lag > agg->max_lag)) - agg->max_lag = lag; - } - } + doStats(agg, skipped, latency, lag); } else { @@ -1780,52 +1787,34 @@ doLog(TState *thread, CState *st, FILE *logfile, instr_time *now, AggVals *agg, * usage), so we don't need to handle this in a special way * (see below). */ - fprintf(logfile, "%ld %d %.0f %.0f %.0f %.0f", + fprintf(logfile, "%ld " INT64_FORMAT " %.0f %.0f %.0f %.0f", agg->start_time, agg->cnt, - agg->sum_latency, - agg->sum2_latency, - agg->min_latency, - agg->max_latency); + agg->latency.sum, + agg->latency.sum2, + agg->latency.min, + agg->latency.max); if (throttle_delay) { fprintf(logfile, " %.0f %.0f %.0f %.0f", - agg->sum_lag, - agg->sum2_lag, - agg->min_lag, - agg->max_lag); + agg->lag.sum, + agg->lag.sum2, + agg->lag.min, + agg->lag.max); if (latency_limit) - fprintf(logfile, " %d", agg->skipped); + fprintf(logfile, " " INT64_FORMAT, agg->skipped); } fputc('\n', logfile); - /* move to the next inteval */ - agg->start_time = agg->start_time + agg_interval; + /* move to the next interval */ + agg->start_time += agg_interval; /* reset for "no transaction" intervals */ - agg->cnt = 0; - agg->skipped = 0; - agg->min_latency = 0; - agg->max_latency = 0; - agg->sum_latency = 0; - agg->sum2_latency = 0; - agg->min_lag = 0; - agg->max_lag = 0; - agg->sum_lag = 0; - agg->sum2_lag = 0; + initStats(agg, 0.0); } /* reset the values to include only the current transaction. */ - agg->cnt = 1; - agg->skipped = skipped ? 1 : 0; - agg->min_latency = latency; - agg->max_latency = latency; - agg->sum_latency = skipped ? 0.0 : latency; - agg->sum2_latency = skipped ? 0.0 : latency * latency; - agg->min_lag = lag; - agg->max_lag = lag; - agg->sum_lag = lag; - agg->sum2_lag = lag * lag; + doStats(agg, skipped, latency, lag); } } else @@ -1835,21 +1824,21 @@ doLog(TState *thread, CState *st, FILE *logfile, instr_time *now, AggVals *agg, /* This is more than we really ought to know about instr_time */ if (skipped) - fprintf(logfile, "%d %d skipped %d %ld %ld", + fprintf(logfile, "%d " INT64_FORMAT " skipped %d %ld %ld", st->id, st->cnt, st->use_file, (long) now->tv_sec, (long) now->tv_usec); else - fprintf(logfile, "%d %d %.0f %d %ld %ld", + fprintf(logfile, "%d " INT64_FORMAT " %.0f %d %ld %ld", st->id, st->cnt, latency, st->use_file, (long) now->tv_sec, (long) now->tv_usec); #else /* On Windows, instr_time doesn't provide a timestamp anyway */ if (skipped) - fprintf(logfile, "%d %d skipped %d 0 0", + fprintf(logfile, "%d "INT64_FORMAT" skipped %d 0 0", st->id, st->cnt, st->use_file); else - fprintf(logfile, "%d %d %.0f %d 0 0", + fprintf(logfile, "%d "INT64_FORMAT" %.0f %d 0 0", st->id, st->cnt, latency, st->use_file); #endif if (throttle_delay) @@ -1858,6 +1847,44 @@ doLog(TState *thread, CState *st, FILE *logfile, instr_time *now, AggVals *agg, } } +/* + * end of transaction statistics + */ +static void +doTxStats(TState *thread, CState *st, instr_time *now, + bool skipped, FILE *logfile, StatsData *agg) +{ + double latency = 0.0, lag = 0.0; + + if ((!skipped || agg_interval) && INSTR_TIME_IS_ZERO(*now)) + INSTR_TIME_SET_CURRENT(*now); + + if (!skipped) + { + /* compute latency & lag if needed */ + latency = INSTR_TIME_GET_MICROSEC(*now) - st->txn_scheduled; + lag = INSTR_TIME_GET_MICROSEC(st->txn_begin) - st->txn_scheduled; + } + + if (progress || throttle_delay || latency_limit) + { + doStats(& thread->stats, skipped, latency, lag); + + /* record over the limit transactions if needed. */ + if (latency_limit && latency > latency_limit) + thread->latency_late++; + } + else + thread->stats.cnt ++; + + if (use_log) + doLog(thread, st, logfile, now, agg, skipped, latency, lag); + + if (per_script_stats) /* mutex? hmmm... these are only statistics */ + doStats(& sql_script[st->use_file].stats, skipped, latency, lag); +} + + /* discard connections */ static void disconnect_all(CState *state, int length) @@ -2254,6 +2281,7 @@ process_commands(char *buf, const char *source, const int lineno) my_commands->command_num = num_commands++; my_commands->type = 0; /* until set */ my_commands->argc = 0; + initSimpleStats(& my_commands->stats); if (*p == '\\') { @@ -2478,7 +2506,7 @@ read_line_from_file(FILE *fd) return NULL; } -static int +static Command ** process_file(char *filename) { #define COMMANDS_ALLOC_NUM 128 @@ -2486,15 +2514,9 @@ process_file(char *filename) Command **my_commands; FILE *fd; int lineno, - index; + index, + alloc_num; char *buf; - int alloc_num; - - if (num_files >= MAX_FILES) - { - fprintf(stderr, "at most %d SQL files are allowed\n", MAX_FILES); - exit(1); - } alloc_num = COMMANDS_ALLOC_NUM; my_commands = (Command **) pg_malloc(sizeof(Command *) * alloc_num); @@ -2506,7 +2528,7 @@ process_file(char *filename) fprintf(stderr, "could not open file \"%s\": %s\n", filename, strerror(errno)); pg_free(my_commands); - return false; + return NULL; } lineno = 0; @@ -2538,13 +2560,11 @@ process_file(char *filename) my_commands[index] = NULL; - sql_files[num_files++] = my_commands; - - return true; + return my_commands; } static Command ** -process_builtin(char *tb, const char *source) +process_builtin(const char *tb, const char *source) { #define COMMANDS_ALLOC_NUM 128 @@ -2598,35 +2618,77 @@ process_builtin(char *tb, const char *source) return my_commands; } +/* Possiby truncate option and return weight */ +static int +getWeight(char *option) +{ + char *sep; + int weight; + + if ((sep = strrchr(option, WSEP))) + { + *sep++ = '\0'; + weight = atoi(sep); + if (weight <= 0) + { + fprintf(stderr, + "weight must be strictly positive, got \"%s\"\n", sep); + exit(1); + } + } + else + weight = 1; + return weight; +} + +static void +addScript(const char *name, Command ** commands, int weight) +{ + if (commands == NULL) + { + fprintf(stderr, "empty commands for %s\n", name); + exit(1); + } + + if (num_scripts >= MAX_SCRIPTS) + { + fprintf(stderr, "at most %d SQL scripts are allowed\n", MAX_SCRIPTS); + exit(1); + } + + sql_script[num_scripts].name = name; + sql_script[num_scripts].weight = weight; + sql_script[num_scripts].commands = commands; + initStats(& sql_script[num_scripts].stats, 0.0); + num_scripts++; +} + +static void +printSimpleStats(char *prefix, SimpleStats *ss) +{ + /* print NaN if no transactions where executed */ + double latency = ss->sum / ss->count; + double stddev = sqrt(ss->sum2 / ss->count - latency*latency); + printf("%s average = %.3f ms\n", prefix, 0.001 * latency); + printf("%s stddev = %.3f ms\n", prefix, 0.001 * stddev); +} + /* print out results */ static void -printResults(int ttype, int64 normal_xacts, int nclients, - TState *threads, int nthreads, - instr_time total_time, instr_time conn_total_time, - int64 total_latencies, int64 total_sqlats, - int64 throttle_lag, int64 throttle_lag_max, - int64 throttle_latency_skipped, int64 latency_late) +printResults(TState *threads, StatsData *total, instr_time total_time, + instr_time conn_total_time, int latency_late) { double time_include, tps_include, tps_exclude; - char *s; time_include = INSTR_TIME_GET_DOUBLE(total_time); - tps_include = normal_xacts / time_include; - tps_exclude = normal_xacts / (time_include - + tps_include = total->cnt / time_include; + tps_exclude = total->cnt / (time_include - (INSTR_TIME_GET_DOUBLE(conn_total_time) / nthreads)); - if (ttype == 0) - s = "TPC-B (sort of)"; - else if (ttype == 2) - s = "Update only pgbench_accounts"; - else if (ttype == 1) - s = "SELECT only"; - else - s = "Custom query"; - - printf("transaction type: %s\n", s); + printf("transaction type: %s\n", + num_scripts == 1? sql_script[0].name: "multiple scripts"); printf("scaling factor: %d\n", scale); printf("query mode: %s\n", QUERYMODE[querymode]); printf("number of clients: %d\n", nclients); @@ -2634,49 +2696,38 @@ printResults(int ttype, int64 normal_xacts, int nclients, if (duration <= 0) { printf("number of transactions per client: %d\n", nxacts); - printf("number of transactions actually processed: " INT64_FORMAT "/" INT64_FORMAT "\n", - normal_xacts, (int64) nxacts * nclients); + printf("number of transactions actually processed: "INT64_FORMAT"/%d\n", + total->cnt, nxacts * nclients); } else { printf("duration: %d s\n", duration); - printf("number of transactions actually processed: " INT64_FORMAT "\n", - normal_xacts); + printf("number of transactions actually processed: "INT64_FORMAT"\n", + total->cnt); } /* Remaining stats are nonsensical if we failed to execute any xacts */ - if (normal_xacts <= 0) + if (total->cnt <= 0) return; if (throttle_delay && latency_limit) - printf("number of transactions skipped: " INT64_FORMAT " (%.3f %%)\n", - throttle_latency_skipped, - 100.0 * throttle_latency_skipped / (throttle_latency_skipped + normal_xacts)); + printf("number of transactions skipped: "INT64_FORMAT" (%.3f %%)\n", + total->skipped, + 100.0 * total->skipped / (total->skipped + total->cnt)); if (latency_limit) - printf("number of transactions above the %.1f ms latency limit: " INT64_FORMAT " (%.3f %%)\n", + printf("number of transactions above the %.1f ms latency limit: %d (%.3f %%)\n", latency_limit / 1000.0, latency_late, - 100.0 * latency_late / (throttle_latency_skipped + normal_xacts)); + 100.0 * latency_late / (total->skipped + total->cnt)); if (throttle_delay || progress || latency_limit) - { - /* compute and show latency average and standard deviation */ - double latency = 0.001 * total_latencies / normal_xacts; - double sqlat = (double) total_sqlats / normal_xacts; - - printf("latency average: %.3f ms\n" - "latency stddev: %.3f ms\n", - latency, 0.001 * sqrt(sqlat - 1000000.0 * latency * latency)); - } + printSimpleStats("latency", & total->latency); else - { /* only an average latency computed from the duration is available */ printf("latency average: %.3f ms\n", - 1000.0 * duration * nclients / normal_xacts); - } + 1000.0 * duration * nclients / total->cnt); if (throttle_delay) - { /* * Report average transaction lag under rate limit throttling. This * is the delay between scheduled and actual start times for the @@ -2684,53 +2735,44 @@ printResults(int ttype, int64 normal_xacts, int nclients, * the database load, or the Poisson throttling process. */ printf("rate limit schedule lag: avg %.3f (max %.3f) ms\n", - 0.001 * throttle_lag / normal_xacts, 0.001 * throttle_lag_max); - } + 0.001 * total->lag.sum / total->cnt, 0.001 * total->lag.max); printf("tps = %f (including connections establishing)\n", tps_include); printf("tps = %f (excluding connections establishing)\n", tps_exclude); - /* Report per-command latencies */ - if (is_latencies) + /* Report per-file data */ + if (per_script_stats) { - int i; + int i; - for (i = 0; i < num_files; i++) + for (i = 0; i < num_scripts; i++) { - Command **commands; + printf("SQL script %d, weight %d: %s\n" + " - "INT64_FORMAT" transactions (%.1f%% of total, tps = %f)\n", + i+1, sql_script[i].weight, sql_script[i].name, + sql_script[i].stats.cnt, + 100.0 * sql_script[i].stats.cnt / total->cnt, + sql_script[i].stats.cnt / time_include); - if (num_files > 1) - printf("statement latencies in milliseconds, file %d:\n", i + 1); - else - printf("statement latencies in milliseconds:\n"); + if (latency_limit) + printf(" - number of transactions skipped: "INT64_FORMAT" (%.3f%%)\n", + sql_script[i].stats.skipped, + 100.0 * sql_script[i].stats.skipped / + (sql_script[i].stats.skipped + sql_script[i].stats.cnt)); - for (commands = sql_files[i]; *commands != NULL; commands++) + printSimpleStats(" - latency", & sql_script[i].stats.latency); + + /* Report per-command latencies */ + if (is_latencies) { - Command *command = *commands; - int cnum = command->command_num; - double total_time; - instr_time total_exec_elapsed; - int total_exec_count; - int t; + Command ** com; - /* Accumulate per-thread data for command */ - INSTR_TIME_SET_ZERO(total_exec_elapsed); - total_exec_count = 0; - for (t = 0; t < nthreads; t++) - { - TState *thread = &threads[t]; + printf(" - per command latencies in ms:\n"); - INSTR_TIME_ADD(total_exec_elapsed, - thread->exec_elapsed[cnum]); - total_exec_count += thread->exec_count[cnum]; - } - - if (total_exec_count > 0) - total_time = INSTR_TIME_GET_MILLISEC(total_exec_elapsed) / (double) total_exec_count; - else - total_time = 0.0; - - printf("\t%f\t%s\n", total_time, command->line); + for (com = sql_script[i].commands; *com != NULL; com++) + printf(" %11.3f %s\n", + 1000.0 * (*com)->stats.sum / (*com)->stats.count, + (*com)->line); } } } @@ -2742,6 +2784,7 @@ main(int argc, char **argv) { static struct option long_options[] = { /* systematic long/short named options */ + {"tpc-b", no_argument, NULL, 'B'}, {"client", required_argument, NULL, 'c'}, {"connect", no_argument, NULL, 'C'}, {"debug", no_argument, NULL, 'd'}, @@ -2752,12 +2795,14 @@ main(int argc, char **argv) {"initialize", no_argument, NULL, 'i'}, {"jobs", required_argument, NULL, 'j'}, {"log", no_argument, NULL, 'l'}, + {"latency-limit", required_argument, NULL, 'L'}, {"no-vacuum", no_argument, NULL, 'n'}, {"port", required_argument, NULL, 'p'}, {"progress", required_argument, NULL, 'P'}, {"protocol", required_argument, NULL, 'M'}, {"quiet", no_argument, NULL, 'q'}, {"report-latencies", no_argument, NULL, 'r'}, + {"rate", required_argument, NULL, 'R'}, {"scale", required_argument, NULL, 's'}, {"select-only", no_argument, NULL, 'S'}, {"skip-some-updates", no_argument, NULL, 'N'}, @@ -2772,25 +2817,19 @@ main(int argc, char **argv) {"unlogged-tables", no_argument, &unlogged_tables, 1}, {"sampling-rate", required_argument, NULL, 4}, {"aggregate-interval", required_argument, NULL, 5}, - {"rate", required_argument, NULL, 'R'}, - {"latency-limit", required_argument, NULL, 'L'}, {NULL, 0, NULL, 0} }; int c; - int nclients = 1; /* default number of simulated clients */ - int nthreads = 1; /* default number of threads */ int is_init_mode = 0; /* initialize mode? */ int is_no_vacuum = 0; /* no vacuum at all before testing? */ int do_vacuum_accounts = 0; /* do vacuum accounts before testing? */ - int ttype = 0; /* transaction type. 0: TPC-B, 1: SELECT only, - * 2: skip update of branches and tellers */ int optindex; - char *filename = NULL; bool scale_given = false; bool benchmarking_option_set = false; bool initialization_option_set = false; + bool internal_script_used = false; CState *state; /* status of clients */ TState *threads; /* array of thread */ @@ -2798,13 +2837,10 @@ main(int argc, char **argv) instr_time start_time; /* start up time */ instr_time total_time; instr_time conn_total_time; - int64 total_xacts = 0; - int64 total_latencies = 0; - int64 total_sqlats = 0; - int64 throttle_lag = 0; - int64 throttle_lag_max = 0; - int64 throttle_latency_skipped = 0; int64 latency_late = 0; + StatsData stats; + int weight; + char *desc; int i; int nclients_dealt; @@ -2850,7 +2886,7 @@ main(int argc, char **argv) state = (CState *) pg_malloc(sizeof(CState)); memset(state, 0, sizeof(CState)); - while ((c = getopt_long(argc, argv, "ih:nvp:dqSNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1) + while ((c = getopt_long(argc, argv, "ih:nvp:dqb:SNc:j:Crs:t:T:U:lf:D:F:M:P:R:L:", long_options, &optindex)) != -1) { switch (c) { @@ -2872,14 +2908,6 @@ main(int argc, char **argv) case 'd': debug++; break; - case 'S': - ttype = 1; - benchmarking_option_set = true; - break; - case 'N': - ttype = 2; - benchmarking_option_set = true; - break; case 'c': benchmarking_option_set = true; nclients = atoi(optarg); @@ -2931,6 +2959,7 @@ main(int argc, char **argv) break; case 'r': benchmarking_option_set = true; + per_script_stats = true; is_latencies = true; break; case 's': @@ -2982,12 +3011,41 @@ main(int argc, char **argv) initialization_option_set = true; use_quiet = true; break; + /* what to run */ + case 'b': + + if (strcmp(optarg, "list") == 0) + { + int i; + fprintf(stdout, "%d builtin scripts: ", N_BUILTIN); + for (i = 0; i < N_BUILTIN; i++) + fprintf(stdout, "%s ", builtin_script[i].name); + fprintf(stdout, "\n"); + exit(0); + } + + weight = getWeight(optarg); + addScript(desc, process_builtin( + find_builtin(optarg, &desc), desc), weight); + benchmarking_option_set = true; + internal_script_used = true; + break; + case 'S': + addScript(desc, process_builtin( + find_builtin("select-only", &desc), desc), 1); + benchmarking_option_set = true; + internal_script_used = true; + break; + case 'N': + addScript(desc, process_builtin( + find_builtin("simple-update", &desc), desc), 1); + benchmarking_option_set = true; + internal_script_used = true; + break; case 'f': + weight = getWeight(optarg); + addScript(optarg, process_file(optarg), weight); benchmarking_option_set = true; - ttype = 3; - filename = pg_strdup(optarg); - if (process_file(filename) == false || *sql_files[num_files - 1] == NULL) - exit(1); break; case 'D': { @@ -3018,9 +3076,9 @@ main(int argc, char **argv) break; case 'M': benchmarking_option_set = true; - if (num_files > 0) + if (num_scripts > 0) { - fprintf(stderr, "query mode (-M) should be specified before any transaction scripts (-f)\n"); + fprintf(stderr, "query mode (-M) should be specified before any transaction scripts (-f or -b)\n"); exit(1); } for (querymode = 0; querymode < NUM_QUERYMODE; querymode++) @@ -3117,6 +3175,23 @@ main(int argc, char **argv) } } + /* set default script if none */ + if (num_scripts == 0 && !initialization_option_set) + { + addScript(desc, process_builtin( + find_builtin("tpcb-like", &desc), desc), 1); + benchmarking_option_set = true; + internal_script_used = true; + } + + /* compute total_weight */ + for (i = 0; i < num_scripts; i++) + total_weight += sql_script[i].weight; + + /* show per script stats if several scripts are used */ + if (!initialization_option_set && num_scripts > 1) + per_script_stats = true; + /* * Don't need more threads than there are clients. (This is not merely an * optimization; throttle_delay is calculated incorrectly below if some @@ -3201,8 +3276,6 @@ main(int argc, char **argv) * changed after fork. */ main_pid = (int) getpid(); - progress_nclients = nclients; - progress_nthreads = nthreads; if (nclients > 1) { @@ -3245,7 +3318,7 @@ main(int argc, char **argv) exit(1); } - if (ttype != 3) + if (internal_script_used) { /* * get the scaling factor that should be same as count(*) from @@ -3329,31 +3402,6 @@ main(int argc, char **argv) INSTR_TIME_SET_CURRENT(start_time); srandom((unsigned int) INSTR_TIME_GET_MICROSEC(start_time)); - /* process builtin SQL scripts */ - switch (ttype) - { - case 0: - sql_files[0] = process_builtin(tpc_b, - "<builtin: TPC-B (sort of)>"); - num_files = 1; - break; - - case 1: - sql_files[0] = process_builtin(select_only, - "<builtin: select only>"); - num_files = 1; - break; - - case 2: - sql_files[0] = process_builtin(simple_update, - "<builtin: simple update>"); - num_files = 1; - break; - - default: - break; - } - /* set up thread data structures */ threads = (TState *) pg_malloc(sizeof(TState) * nthreads); nclients_dealt = 0; @@ -3369,32 +3417,10 @@ main(int argc, char **argv) thread->random_state[0] = random(); thread->random_state[1] = random(); thread->random_state[2] = random(); - thread->throttle_latency_skipped = 0; thread->latency_late = 0; + initStats(& thread->stats, 0.0); nclients_dealt += thread->nstate; - - if (is_latencies) - { - /* Reserve memory for the thread to store per-command latencies */ - int t; - - thread->exec_elapsed = (instr_time *) - pg_malloc(sizeof(instr_time) * num_commands); - thread->exec_count = (int *) - pg_malloc(sizeof(int) * num_commands); - - for (t = 0; t < num_commands; t++) - { - INSTR_TIME_SET_ZERO(thread->exec_elapsed[t]); - thread->exec_count[t] = 0; - } - } - else - { - thread->exec_elapsed = NULL; - thread->exec_count = NULL; - } } /* all clients must be assigned to a thread */ @@ -3437,11 +3463,11 @@ main(int argc, char **argv) #endif /* ENABLE_THREAD_SAFETY */ /* wait for threads and accumulate results */ + initStats(& stats, 0.0); INSTR_TIME_SET_ZERO(conn_total_time); for (i = 0; i < nthreads; i++) { TState *thread = &threads[i]; - int j; #ifdef ENABLE_THREAD_SAFETY if (threads[i].thread == INVALID_THREAD) @@ -3454,21 +3480,13 @@ main(int argc, char **argv) (void) threadRun(thread); #endif /* ENABLE_THREAD_SAFETY */ - /* thread level stats */ - throttle_lag += thread->throttle_lag; - throttle_latency_skipped += threads->throttle_latency_skipped; + /* aggregate thread level stats */ + appendSimpleStats(& stats.latency, & thread->stats.latency); + appendSimpleStats(& stats.lag, & thread->stats.lag); + stats.cnt += thread->stats.cnt; + stats.skipped += thread->stats.skipped; latency_late += thread->latency_late; - if (throttle_lag_max > thread->throttle_lag_max) - throttle_lag_max = thread->throttle_lag_max; INSTR_TIME_ADD(conn_total_time, thread->conn_time); - - /* client-level stats */ - for (j = 0; j < thread->nstate; j++) - { - total_xacts += thread->state[j].cnt; - total_latencies += thread->state[j].txn_latencies; - total_sqlats += thread->state[j].txn_sqlats; - } } disconnect_all(state, nclients); @@ -3484,10 +3502,7 @@ main(int argc, char **argv) */ INSTR_TIME_SET_CURRENT(total_time); INSTR_TIME_SUBTRACT(total_time, start_time); - printResults(ttype, total_xacts, nclients, threads, nthreads, - total_time, conn_total_time, total_latencies, total_sqlats, - throttle_lag, throttle_lag_max, throttle_latency_skipped, - latency_late); + printResults(threads, &stats, total_time, conn_total_time, latency_late); return 0; } @@ -3508,13 +3523,7 @@ threadRun(void *arg) int64 thread_start = INSTR_TIME_GET_MICROSEC(thread->start_time); int64 last_report = thread_start; int64 next_report = last_report + (int64) progress * 1000000; - int64 last_count = 0, - last_lats = 0, - last_sqlats = 0, - last_lags = 0, - last_skipped = 0; - - AggVals aggs; + StatsData last, aggs; /* * Initialize throttling rate target for all of the thread's clients. It @@ -3524,8 +3533,6 @@ threadRun(void *arg) */ INSTR_TIME_SET_CURRENT(start); thread->throttle_trigger = INSTR_TIME_GET_MICROSEC(start); - thread->throttle_lag = 0; - thread->throttle_lag_max = 0; INSTR_TIME_SET_ZERO(thread->conn_time); @@ -3562,16 +3569,17 @@ threadRun(void *arg) INSTR_TIME_SET_CURRENT(thread->conn_time); INSTR_TIME_SUBTRACT(thread->conn_time, thread->start_time); - agg_vals_init(&aggs, thread->start_time); + initStats(&aggs, INSTR_TIME_GET_DOUBLE(thread->start_time)); + last = aggs; /* send start up queries in async manner */ for (i = 0; i < nstate; i++) { CState *st = &state[i]; - Command **commands = sql_files[st->use_file]; + Command **commands = sql_script[st->use_file].commands; int prev_ecnt = st->ecnt; - st->use_file = getrand(thread, 0, num_files - 1); + st->use_file = num_scripts==1? 0: chooseScript(thread); if (!doCustom(thread, st, &thread->conn_time, logfile, &aggs)) remains--; /* I've aborted */ @@ -3599,7 +3607,7 @@ threadRun(void *arg) for (i = 0; i < nstate; i++) { CState *st = &state[i]; - Command **commands = sql_files[st->use_file]; + Command **commands = sql_script[st->use_file].commands; int sock; if (st->con == NULL) @@ -3612,7 +3620,7 @@ threadRun(void *arg) { /* interrupt client which has not started a transaction */ remains--; - st->sleeping = 0; + st->sleeping = false; st->throttling = false; PQfinish(st->con); st->con = NULL; @@ -3705,7 +3713,7 @@ threadRun(void *arg) for (i = 0; i < nstate; i++) { CState *st = &state[i]; - Command **commands = sql_files[st->use_file]; + Command **commands = sql_script[st->use_file].commands; int prev_ecnt = st->ecnt; if (st->con && (FD_ISSET(PQsocket(st->con), &input_mask) @@ -3736,11 +3744,7 @@ threadRun(void *arg) if (now >= next_report) { /* generate and show report */ - int64 count = 0, - lats = 0, - sqlats = 0, - lags = 0, - skipped = 0; + StatsData cur; int64 run = now - last_report; double tps, total_run, @@ -3760,25 +3764,24 @@ threadRun(void *arg) * (If a read from a 64-bit integer is not atomic, you might * get a "torn" read and completely bogus latencies though!) */ - for (i = 0; i < progress_nclients; i++) + initStats(& cur, 0.0); + for (i = 0; i < nthreads; i++) { - count += state[i].cnt; - lats += state[i].txn_latencies; - sqlats += state[i].txn_sqlats; - } - - for (i = 0; i < progress_nthreads; i++) - { - skipped += thread[i].throttle_latency_skipped; - lags += thread[i].throttle_lag; + appendSimpleStats(& cur.latency, & thread[i].stats.latency); + appendSimpleStats(& cur.lag, & thread[i].stats.lag); + cur.cnt += thread[i].stats.cnt; + cur.skipped += thread[i].stats.skipped; } total_run = (now - thread_start) / 1000000.0; - tps = 1000000.0 * (count - last_count) / run; - latency = 0.001 * (lats - last_lats) / (count - last_count); - sqlat = 1.0 * (sqlats - last_sqlats) / (count - last_count); + tps = 1000000.0 * (cur.cnt - last.cnt) / run; + latency = 0.001 * (cur.latency.sum - last.latency.sum) / + (cur.cnt - last.cnt); + sqlat = 1.0 * (cur.latency.sum2 - last.latency.sum2) + / (cur.cnt - last.cnt); stdev = 0.001 * sqrt(sqlat - 1000000.0 * latency * latency); - lag = 0.001 * (lags - last_lags) / (count - last_count); + lag = 0.001 * (cur.lag.sum - last.lag.sum) / + (cur.cnt - last.cnt); fprintf(stderr, "progress: %.1f s, %.1f tps, " @@ -3788,17 +3791,13 @@ threadRun(void *arg) { fprintf(stderr, ", lag %.3f ms", lag); if (latency_limit) - fprintf(stderr, ", " INT64_FORMAT " skipped", - skipped - last_skipped); + fprintf(stderr, ", "INT64_FORMAT" skipped", + cur.skipped - last.skipped); } fprintf(stderr, "\n"); - last_count = count; - last_lats = lats; - last_sqlats = sqlats; - last_lags = lags; + last = cur; last_report = now; - last_skipped = skipped; /* * Ensure that the next report is in the future, in case
-- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers