Re: [HACKERS] Concurrency testing
On Wed, 2009-10-07 at 13:07 -0400, Alvaro Herrera wrote: > David Fetter wrote: > > > I seem to recall that there were some patches to get psql to help with > > such things, but they didn't go in. Time to revive them? > > Yeah, the API they implemented wasn't ideal, so there was some > discussion that ended up with a specification everyone was happy with, > but then nobody got around to implementing it. Feel free to whack that > patch and resubmit ... See in the archives around here: > http://archives.postgresql.org/message-id/8204.1207689...@sss.pgh.pa.us I would hope that Concurrent psql can be revived. There were some issues, but not really major ones. The main requirement is to be able to specify multiple sessions of activity from a single script. I would prefer it if we could do that via psql. If we start inventing new features in other tools we get situation similar to pgbench, which has some cute features, but they aren't in psql, which also has cute features, but different ones. Fragmentation wastes effort. I think Greg's comments are correct but I would say "also correct". There is no reason to have just one test framework. We need as many as we need. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Fri, Oct 09, 2009 at 08:34:55PM +0200, Markus Schiltknecht wrote: > Hi, > > David Fetter wrote: >> 1. Test situations which require more than one connection, possibly >> to different clusters, for example in the cases of Hot Standby and >> Streaming Replication. This further divides into event-based and >> time-based tests. It's this situation I had in mind when I posted, >> and it was inspired by bugs I've seen, the most recent being in >> writeable CTEs. > > Hm.. not sure what you mean by time-base tests, Kick off x. Wait time t1 Kick off y. Wait time t2. etc., etc. > but to me that distinction sounds like regression vs. performance > testing. Or do you think of time-based regression tests? I > certainly favor regression tests that are event-based, as I've run > into too many false errors due to unexpected timings already. Totally agree about regression tests. There might be other kinds of test--stress- or performance tests, for example--that would make more sense time-based. >> 2. Test situations which require *many* connections in order to >> find issues caused, in essence, by load. >> >> Tsung seems well-suited to the latter. > > Agreed. I would also note here, that Tsung seems especially well > suited for performance testing of clustered solutions, as you very > likely have to cluster the testing agents as well to put a decent > load on the SUT. I'd *love* to see PostgreSQL solutions that would outrun a single decent Tsung box, but I suspect that's at least a year or two away. Cheers, David. -- David Fetter http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
Markus Wanner wrote: Hi, Dimitri Fontaine wrote: I even started a DBT2 implementation as tsung modules, back when returning from pgcon 2006: http://tapoueh.org/misc.html#sec7 darcs get http://pgsql.tapoueh.org/dbt2-tsung/ Now *that* is very cool! I definitely need to have a look at it. Have you tried Sun Faban? It allows to manage test runs, compare test results and configurations and features a web interface. Pretty nice as well. But it's all Java, which I like even less than Perl. However, as far as I know, neither of the two support downloading code from a repository and building automatically, before testing. While the buildfarm already does that (partly, testing single patches would be a nice to have, too). It does, sort of. It has a switch called --from-source which lets you specify a local source repository, which can be something you've applied a patch to. Use of this switch disables fetching the source code, checking it for cleanliness, and uploading the results to the central server. It was designed for testing patches. Here's how I use it: pushd srcroot/HEAD cp -a pgsql pgsql.thistest cd pgsql.thistest cvs update patch -p 0 < /path/to/patch_to_test popd ./run_build.pl --verbose --from-source=`pwd`/srcroot/HEAD/pgsql.thistest cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
Hi, Dimitri Fontaine wrote: I even started a DBT2 implementation as tsung modules, back when returning from pgcon 2006: http://tapoueh.org/misc.html#sec7 darcs get http://pgsql.tapoueh.org/dbt2-tsung/ Now *that* is very cool! I definitely need to have a look at it. Have you tried Sun Faban? It allows to manage test runs, compare test results and configurations and features a web interface. Pretty nice as well. But it's all Java, which I like even less than Perl. However, as far as I know, neither of the two support downloading code from a repository and building automatically, before testing. While the buildfarm already does that (partly, testing single patches would be a nice to have, too). Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
Hi, David Fetter wrote: 1. Test situations which require more than one connection, possibly to different clusters, for example in the cases of Hot Standby and Streaming Replication. This further divides into event-based and time-based tests. It's this situation I had in mind when I posted, and it was inspired by bugs I've seen, the most recent being in writeable CTEs. Hm.. not sure what you mean by time-base tests, but to me that distinction sounds like regression vs. performance testing. Or do you think of time-based regression tests? I certainly favor regression tests that are event-based, as I've run into too many false errors due to unexpected timings already. 2. Test situations which require *many* connections in order to find issues caused, in essence, by load. Tsung seems well-suited to the latter. Agreed. I would also note here, that Tsung seems especially well suited for performance testing of clustered solutions, as you very likely have to cluster the testing agents as well to put a decent load on the SUT. Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Thu, 2009-10-08 at 13:00 -0400, Greg Smith wrote: > Perl is the best tool available that runs on the buildfarm, which makes > something written in it useful here, while Tsung is not. Unfortunately. But you're not going to do performance testing on the buildfarm anyway. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
Le 8 oct. 2009 à 19:00, Greg Smith a écrit : Perl is the best tool available that runs on the buildfarm, which makes something written in it useful here, while Tsung is not. Unfortunately. I though we were talking about 2 separate issues here: david's one, for concurrent regression testing, and yours, for load stressing. It does not look like the buildfarm is a good match for the later, so I imagined we could see about bypassing perl this time ( given a serious motive to ). Regards, -- dim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Thu, 2009-10-08 at 13:00 -0400, Greg Smith wrote: > On Thu, 8 Oct 2009, Dimitri Fontaine wrote: > > > sorry guys, perl is not made for concurrent programming, it's not going > > to be easy reaching current tsung level (years of work with ad-hoc > > tools) using a general hobbyist programming language. Woah... I am not much of a Perl fan but to call it a "general hobbyist programming language" certainly identifies the ignorance of the poster. Joshua D. Drake > > -- > * Greg Smith gsm...@gregsmith.com http://www.gregsmith.com Baltimore, MD > -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering If the world pushes look it in the eye and GRR. Then push back harder. - Salamander -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Thu, 8 Oct 2009, Dimitri Fontaine wrote: sorry guys, perl is not made for concurrent programming, it's not going to be easy reaching current tsung level (years of work with ad-hoc tools) using a general hobbyist programming language. Perl is the best tool available that runs on the buildfarm, which makes something written in it useful here, while Tsung is not. Unfortunately. -- * Greg Smith gsm...@gregsmith.com http://www.gregsmith.com Baltimore, MD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Thu, Oct 08, 2009 at 10:28:53AM +0200, Dimitri Fontaine wrote: > http://tsung.erlang-projects.org/ > http://tsung.erlang-projects.org/user_manual.html > http://archives.postgresql.org/pgsql-admin/2008-12/msg00032.php It looks like we're talking about two orthogonal projects here: 1. Test situations which require more than one connection, possibly to different clusters, for example in the cases of Hot Standby and Streaming Replication. This further divides into event-based and time-based tests. It's this situation I had in mind when I posted, and it was inspired by bugs I've seen, the most recent being in writeable CTEs. 2. Test situations which require *many* connections in order to find issues caused, in essence, by load. Tsung seems well-suited to the latter. Cheers, David. -- David Fetter http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
Andrew Dunstan writes: > Last time I built something to drive a huge client load (many > thousands of simultaneous connections to a web app) I did it in highly > threaded Java using HttpUnit from a number of separate client machines. You > wouldn't believe what that managed to do to MySQL on the backend ;-) Last time I've head about that kind of Java stuff, it was a JMeter story. The webapp guys and the JVM admins were so proud that the load test couldn't worry their setup that they were willing to retest using tsung. On a single client rather than the client farm, tsung pushed their setup to an halt under heavy load... and that was with older erlang limits of about 800 connections per node, now it's on the 5 connections per node ballpark IIRC. Just saying :) -- dim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
Greg Smith writes: > The stuff I've been building lately takes "how many sessions at once?" as an > input, and the useful range on that (particularly as we wander further > toward multi-core land) is much higher than you'd want to manage one at a > time manually. I'd rather see a "session simulator" program that picks > among several types of behaviors and executes them, and then you can throw > as many of those as you want into the mix. You can do something like that > with pgbench right now if you pass it multiple files, the tricky part is > figuring out what regression output you should expect. That tool exists and do support PostgreSQL. It's called Tsung and is Open Source: http://tsung.erlang-projects.org/ http://tsung.erlang-projects.org/user_manual.html http://archives.postgresql.org/pgsql-admin/2008-12/msg00032.php It can handle thousands of connections per node and as many nodes as you like, and already includes OS monitoring (adding to its own session / transaction monitoring, of course). It also support SNMP here. Oh, and thinktime too `randomize using a probability distribution (currently exponential)'. I even started a DBT2 implementation as tsung modules, back when returning from pgcon 2006: http://tapoueh.org/misc.html#sec7 darcs get http://pgsql.tapoueh.org/dbt2-tsung/ The thing that tsung isn't made for is tracking the results themselves to validate against regressions. This would have to be done by other means, maybe server side. So Tsung ain't a direct answer to OP [waves to david], or only if we want to add it the checking capability. I'll talk about it with tsung main author. But it answers a lot of the content of this thread, so I figured I'd better talk about it before it's being poorly rewritten in perl. Regards, -- dim PS: erlang is damn easy to learn. PPS: sorry guys, perl is not made for concurrent programming, it's not going to be easy reaching current tsung level (years of work with ad-hoc tools) using a general hobbyist programming language. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Wed, Oct 7, 2009 at 10:41 PM, Jeff Janes wrote: >> I don't really see what's wrong with using Perl modules that are >> likely to be installed most places and easy to obtain where not, if it >> makes writing a test framework much easier. But I also think that we >> should not get bogged down on exactly which tools to use - it seems to >> me the first thing is to find someone who is willing to do the work. >> If someone makes an AWESOME test suite that uses a module which is a >> little too adventurous, we can probably find a way of adjusting it >> after the fact so as to remove the dependency (I fancy myself fairly >> good at this sort of thing, where Perl is concerned). But if we argue >> about tools now, we're just going to discourage anyone from taking a >> stab at it. > > OK, but what would we be taking a stab at? Would it simply be > something like "make dev-check" or "make concurrency-check", which > the build farm would then just invoke, thereby killing two birds with > one stone? Or would it have to be something fancier than just that? > Or am I already too lost in the details? Well, I don't know. I think the first thing would be to figure out what sort of tests we want to run. Taking a stab in the dark, I'm thinking that the point of any sort of concurrency testing must be to check for race conditions, so presumably it's the sort of thing you'd want to have an easy way to run over and over again in a loop. Possibly you would also want to test recovery by killing a backend in the middle somewhere. It may be too much to put into the core tree at all; it might be easier to maintain as a separate project. But I don't have a strong opinion on this. The subject of the thread is "concurrency testing" so it seems to me we ought to start by figuring out exactly what that means and how we could test it. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Wed, Oct 7, 2009 at 5:33 PM, David E. Wheeler wrote: > On Oct 7, 2009, at 5:18 PM, Jeff Janes wrote: > >> I'd much rather live without Test::More and use DBD::Pg, then have >> Test::More but need to open pipes to psql to talk to the database, >> rather than using DBI to do it. But I guess we would need to worry >> about whether we can make DBD::Pg work with the installation being >> tested, rather than finding some other install. > > The test architecture depends on Perl, but not on the DBI. I don't think > that Andrew wants to add any dependencies. Therefore we'd need to use file > handles. If we are going to talk to multiple psql pipes at the same time, and not self-deadlock in the process, we would probably need to use some form of non-blocking IO. Unfortunately, I've often found it isn't as easy as one might hope to do that in a completely portable way. Maybe IPC::Run, but I think that that is non-core also. Are we interested only in simple deterministic concurrency testing: T1 does A, T2 does B, T1 does C and must block until T2 does D, at which point (and not before) T1 must respond with E. (Of course from some recent testing I've done, we would also want "T1 does C and must block either until T2 does D or until X+/-Y microseconds have passed") Or would we want to beat the snot out the database in parallel in a nondeterministic fashion, and checking for self-consistency and non-deadlock? Cheers, Jeff -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Wed, Oct 7, 2009 at 6:01 PM, Robert Haas wrote: > On Wed, Oct 7, 2009 at 8:33 PM, David E. Wheeler wrote: >>> Do we need to restrict ourselves to core? Developers already need >>> flex and bison, which aren't needed when installing from the tarball. >>> Couldn't we also have "make dev-check" that has higher requirements >>> than "make check" does, but does a more thorough job? >> >> flex and bison are not Perl modules. > > True, but so what? Right, my point was only that we already have different levels of requirements for different things. > > I don't really see what's wrong with using Perl modules that are > likely to be installed most places and easy to obtain where not, if it > makes writing a test framework much easier. But I also think that we > should not get bogged down on exactly which tools to use - it seems to > me the first thing is to find someone who is willing to do the work. > If someone makes an AWESOME test suite that uses a module which is a > little too adventurous, we can probably find a way of adjusting it > after the fact so as to remove the dependency (I fancy myself fairly > good at this sort of thing, where Perl is concerned). But if we argue > about tools now, we're just going to discourage anyone from taking a > stab at it. OK, but what would we be taking a stab at? Would it simply be something like "make dev-check" or "make concurrency-check", which the build farm would then just invoke, thereby killing two birds with one stone? Or would it have to be something fancier than just that? Or am I already too lost in the details? Jeff -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
David E. Wheeler wrote: On Oct 7, 2009, at 5:18 PM, Jeff Janes wrote: I'd much rather live without Test::More and use DBD::Pg, then have Test::More but need to open pipes to psql to talk to the database, rather than using DBI to do it. But I guess we would need to worry about whether we can make DBD::Pg work with the installation being tested, rather than finding some other install. The test architecture depends on Perl, but not on the DBI. I don't think that Andrew wants to add any dependencies. Therefore we'd need to use file handles. That's not to say that we couldn't write a nice little interface for it such that the implementation could later change. Well, that's true of the buildfarm. And there are reasons I don't want to use DBI for the buildfarm, mainly to do with its intended role in simulating what a human would do by hand. What we do for the core testing framework is a different question. Nevertheless, a requirement for DBI and DBD::Pg would be a significant escalation of testing prerequisites. Test::More is comparatively modest requirement, and is fairly universal where Perl is installed. And since we'd just be using it to drive psql, we wouldn't be having to decide if a problem we saw was due to a problem in Postgres or a problem in DBD::Pg. If we want something to drive a huge number of clients, I suspect neither of these is a good way to go, and something more custom built would be required. Last time I built something to drive a huge client load (many thousands of simultaneous connections to a web app) I did it in highly threaded Java using HttpUnit from a number of separate client machines. You wouldn't believe what that managed to do to MySQL on the backend ;-) cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Wed, Oct 7, 2009 at 8:33 PM, David E. Wheeler wrote: >> Do we need to restrict ourselves to core? Developers already need >> flex and bison, which aren't needed when installing from the tarball. >> Couldn't we also have "make dev-check" that has higher requirements >> than "make check" does, but does a more thorough job? > > flex and bison are not Perl modules. True, but so what? I don't really see what's wrong with using Perl modules that are likely to be installed most places and easy to obtain where not, if it makes writing a test framework much easier. But I also think that we should not get bogged down on exactly which tools to use - it seems to me the first thing is to find someone who is willing to do the work. If someone makes an AWESOME test suite that uses a module which is a little too adventurous, we can probably find a way of adjusting it after the fact so as to remove the dependency (I fancy myself fairly good at this sort of thing, where Perl is concerned). But if we argue about tools now, we're just going to discourage anyone from taking a stab at it. Just MHO, of course. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Oct 7, 2009, at 5:18 PM, Jeff Janes wrote: I'd much rather live without Test::More and use DBD::Pg, then have Test::More but need to open pipes to psql to talk to the database, rather than using DBI to do it. But I guess we would need to worry about whether we can make DBD::Pg work with the installation being tested, rather than finding some other install. The test architecture depends on Perl, but not on the DBI. I don't think that Andrew wants to add any dependencies. Therefore we'd need to use file handles. That's not to say that we couldn't write a nice little interface for it such that the implementation could later change. Do we need to restrict ourselves to core? Developers already need flex and bison, which aren't needed when installing from the tarball. Couldn't we also have "make dev-check" that has higher requirements than "make check" does, but does a more thorough job? flex and bison are not Perl modules. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Wed, Oct 7, 2009 at 10:38 AM, David Fetter wrote: > On Wed, Oct 07, 2009 at 10:28:08AM -0700, David Wheeler wrote: >> On Oct 7, 2009, at 9:53 AM, David Fetter wrote: >> >>> At the moment, we have no way to test this, although with certain >>> Perl modules, it would be pretty trivial. >> >> No non-core modules necessary. Jus use Test::More and file handles >> opened via pipes to two or more psql sessions. I'd much rather live without Test::More and use DBD::Pg, then have Test::More but need to open pipes to psql to talk to the database, rather than using DBI to do it. But I guess we would need to worry about whether we can make DBD::Pg work with the installation being tested, rather than finding some other install. > > When did Test::More become core? I believe we support back to Perl > 5.6 :/ Do we need to restrict ourselves to core? Developers already need flex and bison, which aren't needed when installing from the tarball. Couldn't we also have "make dev-check" that has higher requirements than "make check" does, but does a more thorough job? Jeff -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Wed, Oct 07, 2009 at 08:06:50PM -0400, Greg Smith wrote: > On Wed, 7 Oct 2009, Alvaro Herrera wrote: > >> I don't find this a compelling argument against concurrent psql. >> Sure there are things you can't do with it, but it doesn't mean >> it's not useful. Are we going to need further tools to find "the >> good concurrent bugs"? No doubt. > > Don't get me wrong, I wasn't arguing against concurrent psql being > useful. Certainly it is. I was just suggesting that the scale of > issues it can be useful for is still pretty limited, and that > accordingly I found my time better spent working on a higher-level > solution that didn't need C-psql anyway. Whether C-psql is > sufficient for what David had in mind I can't say. The kind of stuff I have in mind is regression tests. As with coverage and Dtrace, they might not be the ones that get run by default, but right now, we have no real way in the regression tests to test concurrency issues at all, as far as I know, so it would be good to have some way to catch this stuff. Cheers, David. -- David Fetter http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Wed, 7 Oct 2009, Alvaro Herrera wrote: I don't find this a compelling argument against concurrent psql. Sure there are things you can't do with it, but it doesn't mean it's not useful. Are we going to need further tools to find "the good concurrent bugs"? No doubt. Don't get me wrong, I wasn't arguing against concurrent psql being useful. Certainly it is. I was just suggesting that the scale of issues it can be useful for is still pretty limited, and that accordingly I found my time better spent working on a higher-level solution that didn't need C-psql anyway. Whether C-psql is sufficient for what David had in mind I can't say. -- * Greg Smith gsm...@gregsmith.com http://www.gregsmith.com Baltimore, MD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
Greg Smith wrote: > On Wed, 7 Oct 2009, Alvaro Herrera wrote: > > >Yeah, the API they implemented wasn't ideal, so there was some > >discussion that ended up with a specification everyone was happy with, > >but then nobody got around to implementing it. > > I needed something like this and didn't implement those suggestions > because I thought the whole idea didn't scale up enough. That's > close to the right API to allow more complicated regression tests in > psql itself, but I doubted that would hit real complexity level > needed to find the good concurrent bugs. I don't find this a compelling argument against concurrent psql. Sure there are things you can't do with it, but it doesn't mean it's not useful. Are we going to need further tools to find "the good concurrent bugs"? No doubt. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Wed, 7 Oct 2009, Alvaro Herrera wrote: Yeah, the API they implemented wasn't ideal, so there was some discussion that ended up with a specification everyone was happy with, but then nobody got around to implementing it. I needed something like this and didn't implement those suggestions because I thought the whole idea didn't scale up enough. That's close to the right API to allow more complicated regression tests in psql itself, but I doubted that would hit real complexity level needed to find the good concurrent bugs. The stuff I've been building lately takes "how many sessions at once?" as an input, and the useful range on that (particularly as we wander further toward multi-core land) is much higher than you'd want to manage one at a time manually. I'd rather see a "session simulator" program that picks among several types of behaviors and executes them, and then you can throw as many of those as you want into the mix. You can do something like that with pgbench right now if you pass it multiple files, the tricky part is figuring out what regression output you should expect. We had a quick session on more complicated PG testing at PGCon, and it seems many of us who looked at this from multiple perspectives have settled into the idea that the current regression tests should stay as they are, and the more complicated ones are going to need to be in Perl to keep the code complexity managable but still allow more complicated tests. Another example of a complicted test is something that tests pg_dump/pg_restore, which is also hard to shoehorn into pg_regress. I personally would rather see progress made in that area than trying to hack increasingly difficult tests into the existing regression framework, particular given how changes there impact existing committer's daily workflow. The complexity level of test you need to find the juicy concurrent issues is not one you're going to want to run every time you do "make check". -- * Greg Smith gsm...@gregsmith.com http://www.gregsmith.com Baltimore, MD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Oct 7, 2009, at 10:38 AM, David Fetter wrote: When did Test::More become core? I believe we support back to Perl 5.6 :/ Module::CoreList says 5.006002, though I would have sworn it was added much earlier than that. You could always use Test.pm, I suppose, which has been in core since 5.00405. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Wed, Oct 07, 2009 at 10:28:08AM -0700, David Wheeler wrote: > On Oct 7, 2009, at 9:53 AM, David Fetter wrote: > >> At the moment, we have no way to test this, although with certain >> Perl modules, it would be pretty trivial. > > No non-core modules necessary. Just use Test::More and file handles > opened via pipes to two or more psql sessions. When did Test::More become core? I believe we support back to Perl 5.6 :/ Cheers, David. -- David Fetter http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Oct 7, 2009, at 9:53 AM, David Fetter wrote: At the moment, we have no way to test this, although with certain Perl modules, it would be pretty trivial. No non-core modules necessary. Just use Test::More and file handles opened via pipes to two or more psql sessions. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
David Fetter wrote: > I seem to recall that there were some patches to get psql to help with > such things, but they didn't go in. Time to revive them? Yeah, the API they implemented wasn't ideal, so there was some discussion that ended up with a specification everyone was happy with, but then nobody got around to implementing it. Feel free to whack that patch and resubmit ... See in the archives around here: http://archives.postgresql.org/message-id/8204.1207689...@sss.pgh.pa.us -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Wed, Oct 07, 2009 at 09:17:52AM -0700, David Fetter wrote: > Folks, > > As we move forward, we run into increasingly complex situations under > the general rubric of concurrency. > > What test frameworks are already out there that we can use in our > regression test suite? If there aren't any, how might we build one? Apparently, this wasn't as crystal clear as I thought. Let's imagine you want to test some odd combination of transactions: T1: SET TRANSACTION ISOLATION SERIALIZABLE; T1: SELECT...FOR UPDATE...; T2: UPDATE... T3: COPY... T1: ... At the moment, we have no way to test this, although with certain Perl modules, it would be pretty trivial. I seem to recall that there were some patches to get psql to help with such things, but they didn't go in. Time to revive them? Cheers, David. -- David Fetter http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Concurrency testing
Folks, As we move forward, we run into increasingly complex situations under the general rubric of concurrency. What test frameworks are already out there that we can use in our regression test suite? If there aren't any, how might we build one? Cheers, David. -- David Fetter http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers