Re: [GENERAL] pgbench initialize
Depends on goals of your benchmarking. What are you trying to achieve? Initialization and vacuuming each time will help achieve more consistent best-case numbers (to reduce variance, I'd also destroy cluster completely and clean up hardware, e.g. run fstrim in case of SSD, etc). If you are however after real-world numbers, you may want to initialize once and run pgbench for hours if not days to get full effects of autovaccuum and other possible background processes. On Thu, Dec 15, 2016 at 10:17 PM Anirudh Jayakumar < jayakumar.anir...@gmail.com> wrote: > Hi, > > I have a couple of questions regarding pgbench. I'm new to PostgreSQL and > databases in general, so these might be very trivial questions. > > 1. Is it necessary to initialize pgbench(-i option) before every run to > get consistent results? > > 2. The vacuuming process after initialize consumes time, what are the ill > effects of disabling post-initialize vacuuming? > > Thanks, > Anirudh > smime.p7s Description: S/MIME Cryptographic Signature
Re: [GENERAL] pgbench and scaling
"Are the TPS numbers per pgbench? If so, then you're getting 10x490=4900 TPS system wide, or 20*280=5600 TPS system wide. " Per pgbench. Your explanation makes sense. thanks. -- View this message in context: http://postgresql.nabble.com/pgbench-and-scaling-tp5930891p5931131.html Sent from the PostgreSQL - general mailing list archive at Nabble.com. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench and scaling
On Thu, Nov 17, 2016 at 8:08 PM, Rakesh Kumarwrote: > I noticed that as I scale from 5 to 10 to 20 to 40, the TPS starts falling > almost linearly : > > with 5, TPS was doing 639 > with 10 TPS was down to 490 > with 20 TPS was down to 280 > and so on. Are the TPS numbers per pgbench? If so, then you're getting 10x490=4900 TPS system wide, or 20*280=5600 TPS system wide. If those are total TPS numbers then you should check your disk utilization and CPU utilization and see where your bottleneck is. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench and scaling
I forgot to mention that the db is replicated synchronously. I think that is the culprit. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench tps drop from 5000 to 37 going from localhost to a server 13ms away
On Mon, Jul 27, 2015 at 3:19 AM, Chris Withers ch...@simplistix.co.uk wrote: On 24/07/2015 22:51, Jeff Janes wrote: starting vacuum...end. transaction type: TPC-B (sort of) scaling factor: 1 This is your problem. There is only one row in the pgbench_branch table, and every transaction has to update that one row. This is inherently a seriaized event. Indeed it was! One solution is to just use a large scale on the benchmark so that they upate random pgbench_branch rows, rather than all updating the same row: pgbench -i -s50 With a scale of 1000, everything except the END took roughly the latency time. Interestingly, the END still seems to take more, when threads/clients are really ramped up (100 vs 8). Why would that be? Could it be the fsync time? Presumably your server has a BBU on it, but perhaps the transaction rate at the high scale factor is high enough to overwhelm it. Are you sure you don't see the same timing when you run pgbench locally? Alternatively, you could write a custom file so that all 7 commands are sent down in one packet. How would you restructure the sql so as the make that happen? Just make a file with all the commands in it, and then remove all the newlines from the non-backslash commands so that they are all on the same line (separated by semicolons). Leave the backslash commands on their own lines. Then use the -f switch to pgbench, giving it the file you just made. Also, make sure you give it '-s 1000' as well, because when you use the -f option pgbench does not auto-detect the scale factor. Cheers, Jeff
Re: [GENERAL] pgbench tps drop from 5000 to 37 going from localhost to a server 13ms away
On 24/07/2015 22:51, Jeff Janes wrote: starting vacuum...end. transaction type: TPC-B (sort of) scaling factor: 1 This is your problem. There is only one row in the pgbench_branch table, and every transaction has to update that one row. This is inherently a seriaized event. Indeed it was! One solution is to just use a large scale on the benchmark so that they upate random pgbench_branch rows, rather than all updating the same row: pgbench -i -s50 With a scale of 1000, everything except the END took roughly the latency time. Interestingly, the END still seems to take more, when threads/clients are really ramped up (100 vs 8). Why would that be? Alternatively, you could write a custom file so that all 7 commands are sent down in one packet. How would you restructure the sql so as the make that happen? cheers, Chris
Re: [GENERAL] pgbench tps drop from 5000 to 37 going from localhost to a server 13ms away
That seems to be a large drop. On the other hand 13 ms is also like a very large network latency. On LAN your usually in the sub ms area. So going from e.g. 0.2 ms to 13ms is 65 fold decrease. What is the network toplogy like? Jan Von meinem iPad gesendet Am 24.07.2015 um 18:59 schrieb Chris Withers ch...@simplistix.co.uk: Hi all, I've been doing some lightweight load testing with “pgbench -c8 -j8 -T10” When run locally on the postgres server I've testing, this gives around 5000tps When I do it from a server that has a 13ms ping latency, it drops to 37tps. This is using the default pgbench script, is it to be expected? If so, why? cheers, Chris -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench tps drop from 5000 to 37 going from localhost to a server 13ms away
Am 24.07.2015 um 18:59 schrieb Chris Withers ch...@simplistix.co.uk: Hi all, I've been doing some lightweight load testing with “pgbench -c8 -j8 -T10” When run locally on the postgres server I've testing, this gives around 5000tps When I do it from a server that has a 13ms ping latency, it drops to 37tps. This is using the default pgbench script, is it to be expected? If so, why? Am 24.07.2015 um 20:06 schrieb Jan Lentfer jan.lent...@web.de: That seems to be a large drop. On the other hand 13 ms is also like a very large network latency. On LAN your usually in the sub ms area. So going from e.g. 0.2 ms to 13ms is 65 fold decrease. What is the network toplogy like? Sorry for top posting my first response. Always happens when I am on the iPad. I just checked on my home setup. Ping latency on GBit crossover connection is around 0.3 ms, while pinging localhost is around 0.05 ms. You are at 13ms. So that is a 260 fold decrease, which is in about the same area as what you see with pgbench. Of course with pgbench the actual payload comes into account on top. Jan
Re: [GENERAL] pgbench tps drop from 5000 to 37 going from localhost to a server 13ms away
On 24/07/2015 19:21, Jan Lentfer wrote: I've been doing some lightweight load testing with “pgbench -c8 -j8 -T10” When run locally on the postgres server I've testing, this gives around 5000tps When I do it from a server that has a 13ms ping latency, it drops to 37tps. This is using the default pgbench script, is it to be expected? If so, why? That seems to be a large drop. On the other hand 13 ms is also like a very large network latency. On LAN your usually in the sub ms area. So going from e.g. 0.2 ms to 13ms is 65 fold decrease. What is the network toplogy like? 10G between two colos, a switch at each end. What's interesting is how it goes when adjusting the number for threads/connections. As a baseline, I did one of each. As expected, this gave around 10 tps, (7 statements in the standard file * 13ms latency gives around 0.1s per transaction). This behaviour continued up to -c3 -j3 linearly achieving roughly 10tps per client/thread: starting vacuum...end. transaction type: TPC-B (sort of) scaling factor: 1 query mode: simple number of clients: 3 number of threads: 3 duration: 10 s number of transactions actually processed: 317 latency average: 94.637 ms tps = 31.494417 (including connections establishing) tps = 31.668794 (excluding connections establishing) statement latencies in milliseconds: 0.002016 \set nbranches 1 * :scale 0.000438 \set ntellers 10 * :scale 0.000379 \set naccounts 10 * :scale 0.000489 \setrandom aid 1 :naccounts 0.000404 \setrandom bid 1 :nbranches 0.000413 \setrandom tid 1 :ntellers 0.000470 \setrandom delta -5000 5000 13.061975 BEGIN; 13.174287 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid; 13.127691 SELECT abalance FROM pgbench_accounts WHERE aid = :aid; 14.552413 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid; 14.109375 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid; 13.113028 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP); 13.256063 END; Now, at -c4 and -j4 the problem becomes apparent: starting vacuum...end. transaction type: TPC-B (sort of) scaling factor: 1 query mode: simple number of clients: 4 number of threads: 4 duration: 10 s number of transactions actually processed: 362 latency average: 110.497 ms tps = 35.912951 (including connections establishing) tps = 36.111849 (excluding connections establishing) statement latencies in milliseconds: 0.001917 \set nbranches 1 * :scale 0.000511 \set ntellers 10 * :scale 0.000384 \set naccounts 10 * :scale 0.000525 \setrandom aid 1 :naccounts 0.000406 \setrandom bid 1 :nbranches 0.000472 \setrandom tid 1 :ntellers 0.000483 \setrandom delta -5000 5000 13.063624 BEGIN; 13.170928 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid; 13.122450 SELECT abalance FROM pgbench_accounts WHERE aid = :aid; 16.532138 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid; 28.090450 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid; 13.112207 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP); 13.229887 END; We've dropped down to 9 tps per client/thread, and it's the update statements that are growing in time, here's the worst case: starting vacuum...end. transaction type: TPC-B (sort of) scaling factor: 1 query mode: simple number of clients: 10 number of threads: 10 duration: 10 s number of transactions actually processed: 382 latency average: 261.780 ms tps = 37.310918 (including connections establishing) tps = 37.517192 (excluding connections establishing) statement latencies in milliseconds: 0.001798 \set nbranches 1 * :scale 0.000437 \set ntellers 10 * :scale 0.000385 \set naccounts 10 * :scale 0.000597 \setrandom aid 1 :naccounts 0.000369 \setrandom bid 1 :nbranches 0.000437 \setrandom tid 1 :ntellers 0.000401 \setrandom delta -5000 5000 13.064963 BEGIN; 13.192241 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid; 13.121914 SELECT abalance FROM pgbench_accounts WHERE aid = :aid; 83.994516 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid; 113.638228 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid; 13.133390 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP); 13.288079 END; What on earth could be causing those updates to now take getting on for an order of magnitude more than the latency to the server? cheers, Chris -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench tps drop from 5000 to 37 going from localhost to a server 13ms away
On Fri, Jul 24, 2015 at 12:13 PM, Chris Withers ch...@simplistix.co.uk wrote: On 24/07/2015 19:21, Jan Lentfer wrote: I've been doing some lightweight load testing with “pgbench -c8 -j8 -T10” When run locally on the postgres server I've testing, this gives around 5000tps When I do it from a server that has a 13ms ping latency, it drops to 37tps. This is using the default pgbench script, is it to be expected? If so, why? ... starting vacuum...end. transaction type: TPC-B (sort of) scaling factor: 1 This is your problem. There is only one row in the pgbench_branch table, and every transaction has to update that one row. This is inherently a seriaized event. You trying to overcome the latency by cramming more stuff through the pipe at a time, but what you are cramming through must be done in single-file. One solution is to just use a large scale on the benchmark so that they upate random pgbench_branch rows, rather than all updating the same row: pgbench -i -s50 Alternatively, you could write a custom file so that all 7 commands are sent down in one packet. That way the releasing COMMIT shows up at the same time as the locking UPDATE does, rather than having 2 more round trips between them. But this would violate the spirit of the benchmark, as presumably you are expected to inspect the results of the SELECT statement before proceeding with the rest of the transaction. Or you could write a custom benchmark which more closely resembles whatever your true workload will be. Cheers, Jeff
Re: [GENERAL] pgbench - prevent client from aborting on ERROR
On 04/30/2015 11:36 AM, Nicholson, Brad (Toronto, ON, CA) wrote: Hi, Is there any way to do this? For context, I'm wanting to write a custom script in repeatable read isolation level. If I hit a serializable error, I don't want the client to abort, I want it to continue running transactions. Is that possible? Catch the exception/error and retry: http://www.postgresql.org/docs/9.4/interactive/transaction-iso.html#XACT-REPEATABLE-READ When an application receives this error message, it should abort the current transaction and retry the whole transaction from the beginning. The second time through, the transaction will see the previously-committed change as part of its initial view of the database, so there is no logical conflict in using the new version of the row as the starting point for the new transaction's update. thanks, Brad. -- Adrian Klaver adrian.kla...@aklaver.com -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench
Hi, can you share the plsql procedure to call the query from pgbench i'm running in the same issue cause i have a long query that i want to submit to pgbench There are any news for fix this pgbench issue ? -- View this message in context: http://postgresql.nabble.com/pgbench-tp5773225p5830455.html Sent from the PostgreSQL - general mailing list archive at Nabble.com. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench
A 2013-10-03 17:50, Alvaro Herrera escrigué: Giuseppe Broccolo wrote: The format of the script file has to be one SQL command per line; multiline SQL commands are not supported, and empty lines are ignored. This could bring to errors. Could this be your case? Multiline SQL commands are not supported? Well that sucks, because only BUFSIZ chars are read from each line. In my platform that's 8192, but maybe in Simeó's case it's shorter .. or maybe his query really is longer than 8192 bytes. This smells like a pgbench bug to me. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services Álvaro, you hit the nail on the head! I have seen that in stdio.h BUFSIZ is defined like 1024 in freeBSD 9.0, probably for this reason I can't test this query (1657 characters), but I'm able to test shorter queries. Finally I have done a plsql procedure to call the query from pgbench. Meanwhile I will try to figure out if I can increase this variable without affect the system. Thanks to all Simeó Reig Barcelona (Spain) -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench
On 10/03/2013 06:21 AM, Simeó Reig wrote: Hello I was doing a performance test with pgbench with a pretty long queries and I have the next error: $ pgbench -n -c1 -T 3 -f veins_pgbench.sql pdn Client 0 aborted in state 0: ERROR: syntax error at end of input LINE 1: ...(abc.persones.provincia = abc.poblacions.cod_provinc ^ transaction type: Custom query scaling factor: 1 query mode: simple number of clients: 1 number of threads: 1 duration: 3 s number of transactions actually processed: 0 tps = 0.00 (including connections establishing) tps = 0.00 (excluding connections establishing) I believe pgbench has a very low limit with the queries you can put inside a file with the 't' option. Am I right? How can avoid it ? Well first you say 't' option but show 'T' option, they are different. Second the error is reporting a syntax error in your script, so I would look there first. My best regards Simeó Reig Barcelona (Spain) -- Adrian Klaver adrian.kla...@gmail.com -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench
On 10/03/2013 07:11 AM, Simeó Reig wrote: A 2013-10-03 15:51, Adrian Klaver escrigué: On 10/03/2013 06:21 AM, Simeó Reig wrote: Hello I was doing a performance test with pgbench with a pretty long queries and I have the next error: $ pgbench -n -c1 -T 3 -f veins_pgbench.sql pdn Client 0 aborted in state 0: ERROR: syntax error at end of input LINE 1: ...(abc.persones.provincia = abc.poblacions.cod_provinc ^ transaction type: Custom query scaling factor: 1 query mode: simple number of clients: 1 number of threads: 1 duration: 3 s number of transactions actually processed: 0 tps = 0.00 (including connections establishing) tps = 0.00 (excluding connections establishing) I believe pgbench has a very low limit with the queries you can put inside a file with the 't' option. Am I right? How can avoid it ? Well first you say 't' option but show 'T' option, they are different. Second the error is reporting a syntax error in your script, so I would look there first. Yes, I did I mistake. I would say 'f' option (file option) not 't' option, sorry . But no, there is no mistake with the script, I can do: # psql -d pdn veins_pgbench.sql and it works perfectly thanks Adrian, I'm almost sure that the problem is the query is too long for pgbench (1600 characters) You have not shown the query, but could you be running into the belwo: http://www.postgresql.org/docs/9.3/interactive/pgbench.html The format of a script file is one SQL command per line; multiline SQL commands are not supported. Empty lines and lines beginning with -- are ignored. Script file lines can also be meta commands, which are interpreted by pgbench itself, as described below. Best regards Simeó Reig Barcelona (Spain) -- Adrian Klaver adrian.kla...@gmail.com -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench
A 2013-10-03 15:51, Adrian Klaver escrigué: On 10/03/2013 06:21 AM, Simeó Reig wrote: Hello I was doing a performance test with pgbench with a pretty long queries and I have the next error: $ pgbench -n -c1 -T 3 -f veins_pgbench.sql pdn Client 0 aborted in state 0: ERROR: syntax error at end of input LINE 1: ...(abc.persones.provincia = abc.poblacions.cod_provinc ^ transaction type: Custom query scaling factor: 1 query mode: simple number of clients: 1 number of threads: 1 duration: 3 s number of transactions actually processed: 0 tps = 0.00 (including connections establishing) tps = 0.00 (excluding connections establishing) I believe pgbench has a very low limit with the queries you can put inside a file with the 't' option. Am I right? How can avoid it ? Well first you say 't' option but show 'T' option, they are different. Second the error is reporting a syntax error in your script, so I would look there first. Yes, I did I mistake. I would say 'f' option (file option) not 't' option, sorry . But no, there is no mistake with the script, I can do: # psql -d pdn veins_pgbench.sql and it works perfectly thanks Adrian, I'm almost sure that the problem is the query is too long for pgbench (1600 characters) Best regards Simeó Reig Barcelona (Spain) -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench
A 2013-10-03 16:16, Adrian Klaver escrigué: On 10/03/2013 07:11 AM, Simeó Reig wrote: A 2013-10-03 15:51, Adrian Klaver escrigué: On 10/03/2013 06:21 AM, Simeó Reig wrote: Hello I was doing a performance test with pgbench with a pretty long queries and I have the next error: $ pgbench -n -c1 -T 3 -f veins_pgbench.sql pdn Client 0 aborted in state 0: ERROR: syntax error at end of input LINE 1: ...(abc.persones.provincia = abc.poblacions.cod_provinc ^ transaction type: Custom query scaling factor: 1 query mode: simple number of clients: 1 number of threads: 1 duration: 3 s number of transactions actually processed: 0 tps = 0.00 (including connections establishing) tps = 0.00 (excluding connections establishing) I believe pgbench has a very low limit with the queries you can put inside a file with the 't' option. Am I right? How can avoid it ? Well first you say 't' option but show 'T' option, they are different. Second the error is reporting a syntax error in your script, so I would look there first. Yes, I did I mistake. I would say 'f' option (file option) not 't' option, sorry . But no, there is no mistake with the script, I can do: # psql -d pdn veins_pgbench.sql and it works perfectly thanks Adrian, I'm almost sure that the problem is the query is too long for pgbench (1600 characters) You have not shown the query, but could you be running into the belwo: http://www.postgresql.org/docs/9.3/interactive/pgbench.html The format of a script file is one SQL command per line; multiline SQL commands are not supported. Empty lines and lines beginning with -- are ignored. Script file lines can also be meta commands, which are interpreted by pgbench itself, as described below. I did it, and I read it all ... I'm use to work with postgresql and freeBSD world since too years ago ;-) Thanks again Adrian Simeó Reig Barcelona (Spain) -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench
On 10/03/2013 07:23 AM, Simeó Reig wrote: You have not shown the query, but could you be running into the belwo: http://www.postgresql.org/docs/9.3/interactive/pgbench.html The format of a script file is one SQL command per line; multiline SQL commands are not supported. Empty lines and lines beginning with -- are ignored. Script file lines can also be meta commands, which are interpreted by pgbench itself, as described below. I did it, and I read it all ... I'm use to work with postgresql and freeBSD world since too years ago ;-) Obviously pgbench has issues with the syntax in the file. Without the contents of the file I am just doing some educated guessing. To get to a solution you will need to show your custom file. Thanks again Adrian Simeó Reig Barcelona (Spain) -- Adrian Klaver adrian.kla...@gmail.com -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench
A 2013-10-03 16:40, Adrian Klaver escrigué: On 10/03/2013 07:23 AM, Simeó Reig wrote: You have not shown the query, but could you be running into the belwo: http://www.postgresql.org/docs/9.3/interactive/pgbench.html The format of a script file is one SQL command per line; multiline SQL commands are not supported. Empty lines and lines beginning with -- are ignored. Script file lines can also be meta commands, which are interpreted by pgbench itself, as described below. I did it, and I read it all ... I'm use to work with postgresql and freeBSD world since too years ago ;-) Obviously pgbench has issues with the syntax in the file. Without the contents of the file I am just doing some educated guessing. To get to a solution you will need to show your custom file. Yes you are right, It has an issue with the syntax, because it only reads a portion of the query, and obviously it's not correct if you do it. Unfortunately I can't put the sql here, but I repeat: I could do: psql -d databasename scrip_file.sql and it works perfectly, and It is a one line query. I did it with many queries since today, but I think this is too long Many many thanks for your interest Adrian Simeó Reig Barcelona (Spain) -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench
Il 03/10/2013 16:11, Simeó Reig ha scritto: A 2013-10-03 15:51, Adrian Klaver escrigué: On 10/03/2013 06:21 AM, Simeó Reig wrote: Hello I was doing a performance test with pgbench with a pretty long queries and I have the next error: $ pgbench -n -c1 -T 3 -f veins_pgbench.sql pdn Client 0 aborted in state 0: ERROR: syntax error at end of input LINE 1: ...(abc.persones.provincia = abc.poblacions.cod_provinc ^ transaction type: Custom query scaling factor: 1 query mode: simple number of clients: 1 number of threads: 1 duration: 3 s number of transactions actually processed: 0 tps = 0.00 (including connections establishing) tps = 0.00 (excluding connections establishing) I believe pgbench has a very low limit with the queries you can put inside a file with the 't' option. Am I right? How can avoid it ? Well first you say 't' option but show 'T' option, they are different. Second the error is reporting a syntax error in your script, so I would look there first. Yes, I did I mistake. I would say 'f' option (file option) not 't' option, sorry . But no, there is no mistake with the script, I can do: # psql -d pdn veins_pgbench.sql and it works perfectly thanks Adrian, I'm almost sure that the problem is the query is too long for pgbench (1600 characters) The format of the script file has to be one SQL command per line; multiline SQL commands are not supported, and empty lines are ignored. This could bring to errors. Could this be your case? Giuseppe. -- Giuseppe Broccolo - 2ndQuadrant Italy PostgreSQL Training, Services and Support giuseppe.brocc...@2ndquadrant.it | www.2ndQuadrant.it -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench
Giuseppe Broccolo wrote: The format of the script file has to be one SQL command per line; multiline SQL commands are not supported, and empty lines are ignored. This could bring to errors. Could this be your case? Multiline SQL commands are not supported? Well that sucks, because only BUFSIZ chars are read from each line. In my platform that's 8192, but maybe in Simeó's case it's shorter .. or maybe his query really is longer than 8192 bytes. This smells like a pgbench bug to me. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench
Alvaro Herrera alvhe...@2ndquadrant.com writes: Multiline SQL commands are not supported? Well that sucks, because only BUFSIZ chars are read from each line. In my platform that's 8192, but maybe in Simeó's case it's shorter .. or maybe his query really is longer than 8192 bytes. This smells like a pgbench bug to me. Yeah, I ran into that line-length limit in pgbench a couple months ago. We really oughta remove the restriction; seems like that shouldn't be hard. I'm less enthused about introducing a multiline-command feature; that'd require inventing some escape syntax that's not there now, and making it 100% backwards compatible might be hard. regards, tom lane -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench help
On 12/24/2012 08:50 AM, Atri Sharma wrote: I have two 9.2 servers running in different data directories and installation directories. One of them(the main one) has its bin in the path of my system.The other one does not, hence, I need to use the complete path(/usr/local/pgsql/...). I want to run pgbench on the second server. How should I configure pgbench to use the second server's psql? Hi, I'm by no means an expert in pgbench, but this is client software relying on libpq. You can make it work against the secondary server simply by specifying its port with -p option or PGPORT environment variable. If that matters, to make sure that the right version of libpq is being used, you can use LD_LIBRARY_PATH, something like export LD_LIBRARY_PATH = /usr/local/pgsql/lib ann use if possible the pgbench from the secondary install. Make sure both servers aren't running at the same time :-) Regards, -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench help
On Mon, Dec 24, 2012 at 4:13 PM, Georges Racinet graci...@anybox.fr wrote: On 12/24/2012 08:50 AM, Atri Sharma wrote: I have two 9.2 servers running in different data directories and installation directories. One of them(the main one) has its bin in the path of my system.The other one does not, hence, I need to use the complete path(/usr/local/pgsql/...). I want to run pgbench on the second server. How should I configure pgbench to use the second server's psql? Hi, I'm by no means an expert in pgbench, but this is client software relying on libpq. You can make it work against the secondary server simply by specifying its port with -p option or PGPORT environment variable. If that matters, to make sure that the right version of libpq is being used, you can use LD_LIBRARY_PATH, something like export LD_LIBRARY_PATH = /usr/local/pgsql/lib ann use if possible the pgbench from the secondary install. Make sure both servers aren't running at the same time :-) Thanks a ton, I get it now. Atri -- Regards, Atri l'apprenant -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench help
On 12/24/2012 2:43 AM, Georges Racinet wrote: Make sure both servers aren't running at the same time why? its perfectly OK to ahve severla postgres servers running at once, as long as they are on different port numbers. I generally use 5432, 5433, 5434, etc for this. mostly for development, or for migration, not so much on a production system where performance is important. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench help
On 12/24/2012 12:32 PM, John R Pierce wrote: On 12/24/2012 2:43 AM, Georges Racinet wrote: Make sure both servers aren't running at the same time why? its perfectly OK to ahve severla postgres servers running at once, as long as they are on different port numbers. I generally use 5432, 5433, 5434, etc for this. mostly for development, or for migration, not so much on a production system where performance is important. You're perfectly right. I'm used to have several clusters on the same host (application testing in my case). In this benchmark context, though, I just was being wary that they may interfere, For instance, I suppose that an autovacuum wakeup in one should lower performance results of the other. Sorry it that sounded more general than that. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench help
Sent from my iPad On 24-Dec-2012, at 17:15, Georges Racinet graci...@anybox.fr wrote: On 12/24/2012 12:32 PM, John R Pierce wrote: On 12/24/2012 2:43 AM, Georges Racinet wrote: Make sure both servers aren't running at the same time why? its perfectly OK to ahve severla postgres servers running at once, as long as they are on different port numbers. I generally use 5432, 5433, 5434, etc for this. mostly for development, or for migration, not so much on a production system where performance is important. You're perfectly right. I'm used to have several clusters on the same host (application testing in my case). In this benchmark context, though, I just was being wary that they may interfere, For instance, I suppose that an autovacuum wakeup in one should lower performance results of the other. Sorry it that sounded more general than that. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general Hi, Thanks for the extensive discussion. In my case,vacuum performance plays an important role. Hence, I shall run only one server at a time while profiling. Thanks a ton, Atri -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench out of memory error
Jeff Ross wrote: I'm trying to put a new server on line and I'm having a problem getting any kind of decent performance from it. pgbench yields around 4000 tps until scale and clients both are above 21, then I see the following: NOTICE: ALTER TABLE / ADD PRIMARY KEY will create implicit index pgbench_accounts_pkey for table pgbench_accounts ERROR: out of memory DETAIL: Failed on request of size 67108864. You've got maintenance_work_mem = 240MB, but it looks your OS is not allowing you to allocate more than around 64MB. Have you looked at the active ulimit settings for the accounts involved? The controller cache is set to write thru for all three volumes because tests using dd and bonnie++ show that write thru is twice as fast as write back. I haven't dug into that any more to figure out why. That's bizarre, and you'll never get good pgbench results that way regardless of what dd/bonnie++ say--pgbench does database commits, which is what you need the cache to accelerate, while those two tests don't. But I don't think this is relevant to your immediate problems, because you're running the select-only test so far, which isn't doing writes at all. Regardless, if I got a new system and it performed worse on dd/bonnie++ with the cache turned on, I'd send it back. time pgbench -h $HOST -t 2000 -c $SCALE -S pgbench Your number of transactions here is extremely low. I'd bet you're just measuring startup overhead here. Try using 20,000 per client to start instead and see what happens. On the select-only, you can easily need 1M total transactions to get an accurate reading here. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Pgbench tool download
On Mon, Aug 17, 2009 at 11:56 AM, Chris Barnescompuguruchrisbar...@hotmail.com wrote: I am looking for pgbench. Is there a good source from which I can download the most current version? If you installed from source, look under contrib for the pgbench subdirectory. If you installed from your OS's package repository, try looking for postgres contrib packages (e.g. postgresql-contrib-... in Debian). Josh -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench
Never mind. I found it. On Fri, 2008-08-08 at 12:16 -0400, Heeman Lee wrote: Hi, I can't find pgbench on the package we installed. I built, contrib pkg and installed, but that still didn't generate pgbench anywhere. Any idea where to get pgbench? -- Heeman Lee -- Heeman Lee
Re: [GENERAL] pgbench not setting scale size correctly?
On Fri, Mar 14, 2008 at 2:34 PM, Enrico Sirola [EMAIL PROTECTED] wrote: as you see, the reported scaling factor is 1, but I specified -s 1000, which seems strange... I'm going to recompile it from the sources now. Didn't I get anything or there is a bug somewhere? You must have initialized pgbench with scale 1. While running the tests, it will pick up the scale factor with which it was initialized Thanks, Pavan -- Pavan Deolasee EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench not setting scale size correctly?
Pavan Deolasee [EMAIL PROTECTED] writes: On Fri, Mar 14, 2008 at 2:34 PM, Enrico Sirola [EMAIL PROTECTED] wrote: as you see, the reported scaling factor is 1, but I specified -s 1000, You must have initialized pgbench with scale 1. Yeah, -s is only meaningful when given with -i. Maybe someday we ought to fix pgbench to complain if you try to set it at other times. regards, tom lane -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench not setting scale size correctly?
On Fri, 14 Mar 2008, Tom Lane wrote: Pavan Deolasee [EMAIL PROTECTED] writes: You must have initialized pgbench with scale 1. Yeah, -s is only meaningful when given with -i. Maybe someday we ought to fix pgbench to complain if you try to set it at other times. You have to pass -s in to the actual run if you're specifying your own custom script(s) using -f and you want the :scale variable to be defined. But if you're running one of the internal scripts, it just looks in the database instead, overriding whatever you pass. The way the option parsing code is done would make complaining in the case where your parameter is ignored a bit of a contortion. The part that detects based on the database is after all the other parsing because the connection has to be brought up first. This is somewhat documented as of 8.3 (I think I submitted that); see the notes on -s in http://www.postgresql.org/docs/current/static/pgbench.html#PGBENCH-RUN-OPTIONS That could be clearer though, it makes it sound like it's a display only thing where it's actually quite functional if your custom script uses scale. I have another pgbench doc patch I'm working on, when I get that done I'll correct this as well. -- * Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench not setting scale size correctly?
Greg Smith [EMAIL PROTECTED] writes: On Fri, 14 Mar 2008, Tom Lane wrote: Yeah, -s is only meaningful when given with -i. Maybe someday we ought to fix pgbench to complain if you try to set it at other times. You have to pass -s in to the actual run if you're specifying your own custom script(s) using -f and you want the :scale variable to be defined. Right, I knew that at one time ;-) The way the option parsing code is done would make complaining in the case where your parameter is ignored a bit of a contortion. The part that detects based on the database is after all the other parsing because the connection has to be brought up first. Yeah. But couldn't we have that part issue a warning if -s had been set on the command line? regards, tom lane -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench not setting scale size correctly?
Tom Lane wrote: Greg Smith [EMAIL PROTECTED] writes: On Fri, 14 Mar 2008, Tom Lane wrote: Yeah, -s is only meaningful when given with -i. Maybe someday we ought to fix pgbench to complain if you try to set it at other times. You have to pass -s in to the actual run if you're specifying your own custom script(s) using -f and you want the :scale variable to be defined. Right, I knew that at one time ;-) The way the option parsing code is done would make complaining in the case where your parameter is ignored a bit of a contortion. The part that detects based on the database is after all the other parsing because the connection has to be brought up first. Yeah. But couldn't we have that part issue a warning if -s had been set on the command line? regards, tom lane I was wondering why the -s would not rescale the data? IMHO a warning would be fine
Re: [GENERAL] pgbench not setting scale size correctly?
Justin [EMAIL PROTECTED] writes: I was wondering why the -s would not rescale the data? That would involve re-initializing the table contents. If that's what you want, use -i. regards, tom lane -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench not setting scale size correctly?
On Fri, 14 Mar 2008, Justin wrote: I was wondering why the -s would not rescale the data? First, you don't know how to rescale the data if someone is passing in a custom script. More importantly, people don't expect the benchmark tool to change things in tables unless specifically requested. I once made the unfortunate mistake of running 'pgbench -i' against the wrong database, one that just happened to have a table named 'accounts' in it, only to watch that table get destroyed. It would be unwise to make that behavior easier to trigger. pgbench is a fairly flexible tool that supports creating simple scripted tests for a variety of purpose. It just so happens that most people only ever use the default tests where it seems things could be simplified. They could, but it would take some of the flexibility out as well. -- * Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench not setting scale size correctly?
Tom Lane wrote: Justin [EMAIL PROTECTED] writes: I was wondering why the -s would not rescale the data? That would involve re-initializing the table contents. If that's what you want, use -i. regards, tom lane thanks
Re: [GENERAL] pgbench not setting scale size correctly?
Greg Smith wrote: On Fri, 14 Mar 2008, Justin wrote: I was wondering why the -s would not rescale the data? First, you don't know how to rescale the data if someone is passing in a custom script. More importantly, people don't expect the benchmark tool to change things in tables unless specifically requested. I once made the unfortunate mistake of running 'pgbench -i' against the wrong database, one that just happened to have a table named 'accounts' in it, only to watch that table get destroyed. It would be unwise to make that behavior easier to trigger. pgbench is a fairly flexible tool that supports creating simple scripted tests for a variety of purpose. It just so happens that most people only ever use the default tests where it seems things could be simplified. They could, but it would take some of the flexibility out as well. -- * Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD OK, what if it warns on the -s is set when missing -i and when ( -i and -f is set in the same command) ??? -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] pgbench
pgbench comes with PostgreSQL source code distribution and must be compiled under PostgreSQL source tree. I'm not familiar with MacOS but I guess there may be pre-compiled package. Anyone can help him? -- Tatsuo Ishii SRA OSS, Inc. Japan I have just received my new MacBook (1.83Gz. Duo) and wanted to do some benchmark testing of Postgres using your Pgbench, however I ma getting errors while compiling. I could probably fix some of these (define TRUE RAND_MAX) I am not sure what RAND_MAX is on other systems and as far as the exit(1)? again not sure why. Any help would be appreciated. here is the output: macbook:~/Desktop/pgbench-1.1 jeffruss$ make gcc -O2 -I/usr/local/pgsql/include -c -o pgbench.o pgbench.c pgbench.c:24:22: error: postgres.h: No such file or directory pgbench.c: In function 'getrand': pgbench.c:94: error: 'RAND_MAX' undeclared (first use in this function) pgbench.c:94: error: (Each undeclared identifier is reported only once pgbench.c:94: error: for each function it appears in.) pgbench.c: In function 'doOne': pgbench.c:131: error: 'TRUE' undeclared (first use in this function) pgbench.c:216: warning: incompatible implicit declaration of built-in function 'strcpy' pgbench.c: In function 'doSelectOnly': pgbench.c:267: error: 'TRUE' undeclared (first use in this function) pgbench.c: In function 'init': pgbench.c:355: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c:362: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c:370: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c:378: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c:389: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c:397: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c:407: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c:415: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c:420: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c:426: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c:434: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c: In function 'main': pgbench.c:523: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c:532: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c:537: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c:545: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c:552: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c:557: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c:565: warning: assignment makes pointer from integer without a cast pgbench.c:573: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c:588: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c:595: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c:600: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c:609: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c:616: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c:626: error: 'uint' undeclared (first use in this function) pgbench.c:626: error: parse error before 'tv1' pgbench.c:637: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c:659: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c:671: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c:688: warning: incompatible implicit declaration of built-in function 'exit' pgbench.c:695: warning: incompatible implicit declaration of built-in function 'exit' make: *** [pgbench.o] Error 1 ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster