Re: [HACKERS] Quality and Performance
On Tue, 27 Nov 2007, Simon Riggs wrote: My vision for that is a set of tests that test very specific aspects of code, much the same way as the regression tests attempt feature coverage. Examples would be - 1 INSERTs - 1 INSERTs using multi-VALUEs clauses - 10 rows inserted by COPY - 10 rows inserted by CTAS In addition to these, there's actually a nice set of sample primitive operations to test included in the Advanced transactional section of sysbench: http://sysbench.sourceforge.net/docs/#database_mode I have code that does this sort of thing, including tests at multiple client loads, that even spits out HTML for the results. It's based on pgbench and I sent out hundreds of results generated by it when I was doing the background writer optimization. The results are thoroughly boring on just my system, and I'm just waiting for a larger server to be available before it's worth my trouble to revive that project. Once such a system becomes available, I could have a basic set of tests like these ready the day after I get a login. That's not to say what you're talking about isn't needed, because using pgbench introduces some limitations and buildfarm integration is a whole different topic. But I have made a first move here and only need the hardware to become available before I can produce something useful with it. -- * Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
[HACKERS] Quality and Performance
Every release we seem to have the same debates about performance issues. In 8.0 we shipped knowing that bgwriter had serious deficiencies, plus had no way of logging SQL statements for performance tuning. In 8.2 we even ended up tweaking the planner *after* release. What I don't understand is all the words about quality, yet we don't seem to include performance as part of that. Performance always seems to be a feature that can be left until the next release and it's never the right time to fix it. I would hope to persuade all that Performance is an integral part of Quality, not a hindrance to it. I've never worked on a software project where either the Users or the Sponsors said don't worry about performance, it can wait, but I really love the way you coded that. Quality is very, very high with Postgres, but we also need to include performance as one of the Top Level concerns *and* do that without dropping the ball on other concerns. That clearly takes time and effort to balance those concerns. We obviously need a performance build farm and I think everyone accepts that. We just need to do it, so that's a given and is something I hope to be involved in. What I would really like to persuade everybody is that performance needs specific attention. Once we've finished integrating the code, we're in Beta and changes seem to be more difficult then. We must give time and attention both to measuring performance and to fixing the things we find. Sure we've done a lot of that, and I've been very happy with that, but recent events make me think we have lapsed back into thinking that performance is a threat to quality. I'd love to hear people say loud and clear that performance matters and we can't ship when we know about fixable performance holes. Please can we clear some space in the next release schedule for performance, plus give some credence to the thought that performance issues rate our attention just as much as other kinds of bugs? Maybe we should give each Beta a name, such as Initial Beta, Performance Beta, Usability Beta as a way of encouraging folk to focus onto particular aspects of quality at what we consider to be appropriate times to do so. Not sure whether thats a good idea, but I'd love to hear about ways to include performance as one of the essential behaviours of PostgreSQL. Your thoughts are welcome, -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Quality and Performance
On Tue, Nov 27, 2007 at 05:32:49PM +, Simon Riggs wrote: What I would really like to persuade everybody is that performance needs specific attention. [. . .] Your thoughts are welcome, Well, one thing that might help is something of the specifics you mention. I remember mentioning to Jan not long after he started at Afilias that we occasionally saw strange behaviour that looked like lock up. He was slightly incredulous, and I didn't have time to build a repeatable test case. So it was in the context of testing Slony that he discovered the dual pains of buffer shuffling and checkpoint storms; this is part of what led him to work on those problems in 8.0. The key was to state, at the outset, Here is the problem I want to fix. By stating precisely and specifically what is to be fixed, the issue moves from performance needs to a feature that can be implemented. Perhaps now is the time to list some specific performance areas you want to fix up? A -- Andrew Sullivan Old sigs will return after re-constitution of blue smoke ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Quality and Performance
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Tue, 27 Nov 2007 17:32:49 + Simon Riggs [EMAIL PROTECTED] wrote: Maybe we should give each Beta a name, such as Initial Beta, Performance Beta, Usability Beta as a way of encouraging folk to focus onto particular aspects of quality at what we consider to be appropriate times to do so. Not sure whether thats a good idea, but I'd love to hear about ways to include performance as one of the essential behaviours of PostgreSQL. Your thoughts are welcome, Well I think that we do take performance into account. I agree that we should *never* have a regression in performance from release to release, which is what I believe has inspired this thread. However if you look at a lot of the items that have gone into this release they were all about performance: HOT -- has a direct correlation to performance because without it we vacuum more. Not only more, but in a lot of cases it resolves a serious PostgreSQL limitation for high velocity tables. Phantom Xid - less vacuuming The sequential scan thing Jeff Davis did The reduced size of the tuple headers Sincerely, Joshua D. Drake - -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240 PostgreSQL solutions since 1997 http://www.commandprompt.com/ UNIQUE NOT NULL Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL Replication: http://www.commandprompt.com/products/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFHTF0XATb/zqfZUUQRAul2AKCfKToFWWkdNWYZTCLmjJJeDQpysQCfTKnI kURZp3SdAoyRLScxG5PizDo= =PTcF -END PGP SIGNATURE- ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] Quality and Performance
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Tue, 27 Nov 2007 18:18:52 + Simon Riggs [EMAIL PROTECTED] wrote: On Tue, 2007-11-27 at 10:08 -0800, Joshua D. Drake wrote: Simon Riggs [EMAIL PROTECTED] wrote: Agreed. I either initiated or assisted with most of those items; but that's not really my point, however because those were planned performance features. My thinking was about how we handle the last minute attention to detail that ensures we have performance everywhere, not just on the main features. Well that is certainly a valid point. Unfortunately that is also a lot of work :). We would have to have some benchmark that actually attributes itself to common real world load (even the very simple real world load like was just fixed). I would also be happy to put some resources behind this. Sincerely, Joshua D. Drake - -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240 PostgreSQL solutions since 1997 http://www.commandprompt.com/ UNIQUE NOT NULL Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL Replication: http://www.commandprompt.com/products/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFHTGEoATb/zqfZUUQRAgk8AKCLNFPpkEqKF44PyXI9D7k5ynZDSgCfbNNm 5JcjMi1+ZxWX4CzQn4y0Bdk= =hvTR -END PGP SIGNATURE- ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Quality and Performance
Simon Riggs wrote: We obviously need a performance build farm and I think everyone accepts that. We just need to do it, so that's a given and is something I hope to be involved in. It's on my list ... Had I but world enough and time ... Performance testing can be bolted onto the exiting buildfarm as an option. However, performance test machines have some requirements that pure functional/build test machines don't have: especially stability. A standard buildfarm client can be put on almost any machine and run happily. My main workstation runs four buildfarm members including three in a VM, and I never notice any impact. But a performance test machine probably needs to be dedicated to just that function. And at least some members of the performance test machines would need to be higher end machines. The number of people who can afford such resources is much lower than those who can run a relatively low impact simple buildfarm member. Maybe we also need to talk about running clients elsewhere for performance testing too. We also need to talk about what would be a good set of tests to run. One useful thing this would buy us is a time series of test results so we could easily see sudden degradations in performance. It must have been annoying trying to triangulate performance dropoff recently. cheers andrew ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] Quality and Performance
Joshua D. Drake wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Tue, 27 Nov 2007 17:32:49 + Simon Riggs [EMAIL PROTECTED] wrote: Maybe we should give each Beta a name, such as Initial Beta, Performance Beta, Usability Beta as a way of encouraging folk to focus onto particular aspects of quality at what we consider to be appropriate times to do so. Not sure whether thats a good idea, but I'd love to hear about ways to include performance as one of the essential behaviours of PostgreSQL. Well I think that we do take performance into account. I agree that we should *never* have a regression in performance from release to release, which is what I believe has inspired this thread. Hmm. I have developed several features that have driven performance down. Autovacuum enabled by default for one. IIRC the SELECT FOR SHARE stuff (tuple-level share locks) also hurt performance. Savepoints required enlarging tuple headers, which also hurt performance. In all cases we have gotten some other benefit, be it reduced administrative pain, or reduced lock contention, or a new feature. (In fact I think most performance drops have been my fault.) -- Alvaro Herrerahttp://www.advogato.org/person/alvherre Some men are heterosexual, and some are bisexual, and some men don't think about sex at all... they become lawyers (Woody Allen) ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Quality and Performance
On Tue, 2007-11-27 at 10:08 -0800, Joshua D. Drake wrote: Simon Riggs [EMAIL PROTECTED] wrote: Maybe we should give each Beta a name, such as Initial Beta, Performance Beta, Usability Beta as a way of encouraging folk to focus onto particular aspects of quality at what we consider to be appropriate times to do so. Not sure whether thats a good idea, but I'd love to hear about ways to include performance as one of the essential behaviours of PostgreSQL. Your thoughts are welcome, Well I think that we do take performance into account. I agree that we should *never* have a regression in performance from release to release, which is what I believe has inspired this thread. However if you look at a lot of the items that have gone into this release they were all about performance: Agreed. I either initiated or assisted with most of those items; but that's not really my point, however because those were planned performance features. My thinking was about how we handle the last minute attention to detail that ensures we have performance everywhere, not just on the main features. -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Quality and Performance
Alvaro Herrera [EMAIL PROTECTED] writes: Joshua D. Drake wrote: Well I think that we do take performance into account. I agree that we should *never* have a regression in performance from release to release, which is what I believe has inspired this thread. Hmm. I have developed several features that have driven performance down. Even changes that are not feature additions but intended solely to improve performance may have corner cases where they are losses rather than wins. I think *never* have a regression in performance is not only pie-in-the-sky but would be a bad policy to adopt, because it would mean for instance that we couldn't intentionally optimize common cases at the expense of uncommon ones. However, I think everybody agrees that getting blindsided by unexpected performance dropoffs is a bad thing. We really need to reinstitute the sort of daily (or near-daily) performance tracking that Mark Wong used to be doing, and extend it to cover a wider variety of test cases than just DBT-2. As an example, I'll bet that this issue of operator lookup speed would never have been visible at all in DBT-2. regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Quality and Performance
On Tue, 2007-11-27 at 13:54 -0500, Tom Lane wrote: However, I think everybody agrees that getting blindsided by unexpected performance dropoffs is a bad thing. We really need to reinstitute the sort of daily (or near-daily) performance tracking that Mark Wong used to be doing, and extend it to cover a wider variety of test cases than just DBT-2. As an example, I'll bet that this issue of operator lookup speed would never have been visible at all in DBT-2. Yeh, we need multiple large benchmarks run on a regular basis. My understanding is the community has two 8-core servers to run benchmarks on, but I'd quite like to have some details on where these are at. One is likely to be running RHEL, one Solaris. We also need performance regression tests, which is a slightly different thing even if they do sound similar. -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Quality and Performance
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Tue, 27 Nov 2007 20:32:57 + Simon Riggs [EMAIL PROTECTED] wrote: On Tue, 2007-11-27 at 13:54 -0500, Tom Lane wrote: However, I think everybody agrees that getting blindsided by unexpected performance dropoffs is a bad thing. We really need to reinstitute the sort of daily (or near-daily) performance tracking that Mark Wong used to be doing, and extend it to cover a wider variety of test cases than just DBT-2. As an example, I'll bet that this issue of operator lookup speed would never have been visible at all in DBT-2. Yeh, we need multiple large benchmarks run on a regular basis. My understanding is the community has two 8-core servers to run benchmarks on, but I'd quite like to have some details on where these are at. One is likely to be running RHEL, one Solaris. The RHEL one as I know it, is the MyYearbook donated one. We are currently unaware of the status of that machine except to say it is currently running Gentoo. I don't know the status of the Solaris machine except that I think we had IO issues with it. Sincerely, Joshua D. Drake We also need performance regression tests, which is a slightly different thing even if they do sound similar. - -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240 PostgreSQL solutions since 1997 http://www.commandprompt.com/ UNIQUE NOT NULL Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL Replication: http://www.commandprompt.com/products/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFHTH+7ATb/zqfZUUQRAhlYAJ0QqG5CzDmQfi2ynj/i9xJoe1nWUACeOyJQ gSOnZhVhp61HbOlfRjfDzvM= =MEIW -END PGP SIGNATURE- ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] Quality and Performance
On Tue, 2007-11-27 at 13:32 -0500, Andrew Dunstan wrote: We also need to talk about what would be a good set of tests to run. I think we should develop a series of performance regression tests that can be run as an option on the buildfarm. We'd want a separate page for that with graphs etc, as you suggest. My vision for that is a set of tests that test very specific aspects of code, much the same way as the regression tests attempt feature coverage. Examples would be - 1 INSERTs - 1 INSERTs using multi-VALUEs clauses - 10 rows inserted by COPY - 10 rows inserted by CTAS We would need a way to compare results between releases, so we can see which aspects have regressed/improved, just as we have with the buildfarm. That will also be food for release notes, where we can mention all actions that are 5% faster, or anything we must regrettably report as being slower. Sounds like it's waiting on somebody to make the first move, so maybe I should do that, then let everybody else chip into the framework. Should we do this as part of core, or as a separate pgfoundry project? -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Quality and Performance
On Tue, 2007-11-27 at 12:36 -0800, Joshua D. Drake wrote: The RHEL one as I know it, is the MyYearbook donated one. We are currently unaware of the status of that machine except to say it is currently running Gentoo. I don't know the status of the Solaris machine except that I think we had IO issues with it. Might I enquire who has these machines, so I can ask them how can I get access to them? Are they really Community systems? In what sense? -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Quality and Performance
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Tue, 27 Nov 2007 21:00:03 + Simon Riggs [EMAIL PROTECTED] wrote: On Tue, 2007-11-27 at 12:36 -0800, Joshua D. Drake wrote: The RHEL one as I know it, is the MyYearbook donated one. We are currently unaware of the status of that machine except to say it is currently running Gentoo. I don't know the status of the Solaris machine except that I think we had IO issues with it. Might I enquire who has these machines, so I can ask them how can I get access to them? Are they really Community systems? In what sense? They are community systems in the sense that JoshB told the community Sun donated a box and MyYearbook said they donated a box. MyYearbook had stated that we (cmd) were going to manage the MyYearbook box but nothing has come of it as of yet. I haven't heard anything on the Sun machine in a *long* time. My guess is that everyone is busy. So I have a question :)... The community does have money. We have IO available, what we don't have is a machine. Would the community be interested in purchasing a box? We could fairly reasonably get an HP or Dell (the new Dell's aren't nearly as bad as they used to be) or Sun 8 core box with reasonable ram and just a raid 1 for the OS. Then use the external IO we have available with that machine. Joshua D. Drake - -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240 PostgreSQL solutions since 1997 http://www.commandprompt.com/ UNIQUE NOT NULL Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL Replication: http://www.commandprompt.com/products/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFHTIh5ATb/zqfZUUQRArJ1AJ4jf6S12CY4duUTYzlSQlF+YTYRvwCgmu/i Dj36cknbGDX2Bzh6rmIkNw8= =jl1/ -END PGP SIGNATURE- ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Quality and Performance
On Tue, 2007-11-27 at 15:33 -0300, Alvaro Herrera wrote: Joshua D. Drake wrote: I agree that we should *never* have a regression in performance from release to release, which is what I believe has inspired this thread. Hmm. I have developed several features that have driven performance down. I think performance reductions as a result of additional functionality are acceptable, but we should aim to minimise them. It's the small things that crop up along the way that must be fixed, with the same vigour we fix other bugs. -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Quality and Performance
Simon Riggs wrote: On Tue, 2007-11-27 at 13:32 -0500, Andrew Dunstan wrote: We also need to talk about what would be a good set of tests to run. Sounds like it's waiting on somebody to make the first move, so maybe I should do that, then let everybody else chip into the framework. If you start with a set of tests and send it to me I will start work on a benchmarking step in the buildfarm client. Should we do this as part of core, or as a separate pgfoundry project? Core, please. This is mainline -hackers material. cheers andrew ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] Quality and Performance
Andrew Dunstan [EMAIL PROTECTED] writes: Simon Riggs wrote: Should we do this as part of core, or as a separate pgfoundry project? Core, please. This is mainline -hackers material. Huh? The buildfarm isn't in core, why would a performfarm be? regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Quality and Performance
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Tue, 27 Nov 2007 21:00:03 + Simon Riggs [EMAIL PROTECTED] wrote: On Tue, 2007-11-27 at 12:36 -0800, Joshua D. Drake wrote: The RHEL one as I know it, is the MyYearbook donated one. We are currently unaware of the status of that machine except to say it is currently running Gentoo. I don't know the status of the Solaris machine except that I think we had IO issues with it. Might I enquire who has these machines, so I can ask them how can I get access to them? Are they really Community systems? In what sense? We just spoke with MyYearbook and my suspicions were correct. Everyone is just busy. Next ETA on that machine is 3 weeks (on the outside).. so expect the beginning of the year. Joshua D. Drake - -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240 PostgreSQL solutions since 1997 http://www.commandprompt.com/ UNIQUE NOT NULL Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL Replication: http://www.commandprompt.com/products/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFHTKN4ATb/zqfZUUQRAoTiAKCC6+RNPFihWyos4s7vCmwt2K3C1gCfdFkV z0ht4nKhelQ+UUwrHnMLXkA= =2JWW -END PGP SIGNATURE- ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Quality and Performance
On Nov 27, 2007 11:45 PM, Andrew Dunstan [EMAIL PROTECTED] wrote: If you start with a set of tests and send it to me I will start work on a benchmarking step in the buildfarm client. Are you sure it shouldn't be a separate client? I don't think neither the prerequisites nor the results wanted have something in common with the build farm. Another idea is that we should find a way to let people run their specific benchmarks and use the client to report their results. Some people may want to run read only benchmarks, some I/O heavy benchmarks, and others benchmarks which runs for more than a day. -- Guillaume ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Quality and Performance
Tom Lane wrote: Andrew Dunstan [EMAIL PROTECTED] writes: Simon Riggs wrote: Should we do this as part of core, or as a separate pgfoundry project? Core, please. This is mainline -hackers material. Huh? The buildfarm isn't in core, why would a performfarm be? It's the tests I think belong in core, not the farm software. Currently buildfarm performs functionality tests that are also in core. cheers andrew ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Quality and Performance
On Nov 27, 2007 7:32 PM, Andrew Dunstan [EMAIL PROTECTED] wrote: But a performance test machine probably needs to be dedicated to just that function. And at least some members of the performance test machines would need to be higher end machines. The number of people who can afford such resources is much lower than those who can run a relatively low impact simple buildfarm member. As I already indicated it in another thread, we will dedicate 7 servers donated by Continuent to exactly that in a near future (they are waiting in their boxes atm). So we will have dedicated boxes to run benchmarks night day. They will be accessible to the community to set up as much benchmarks as necessary and I'll help to set up them and make sure everything is OK. They are not high end servers as the MyYearBook or Sun ones but they are pretty decent. They are all i386 so we'll have to get other clients but I think we can set up a variety of benchmarks to get them busy and produce valuable results. If we can get a benchmark client/results server infrastructure by the end of january, it will be right on time. I'm pretty sure all vendors of PostgreSQL are aware that a bench farm is really important right now so I'm pretty sure we'll get more servers if we can get something as easy to setup as the build farm. -- Guillaume ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Quality and Performance
Andrew, It's the tests I think belong in core, not the farm software. Currently buildfarm performs functionality tests that are also in core. Jignesh and I were talking about writing a Pole Position-style test which measures peformance on each of a couple dozen specific operations. There are limits to what we could ship with PostgreSQL ... DW operations aren't really testable without 18 hours to generate data ... but we could test a lot of things. -- --Josh Josh Berkus PostgreSQL @ Sun San Francisco ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Quality and Performance
Josh Berkus wrote: Andrew, It's the tests I think belong in core, not the farm software. Currently buildfarm performs functionality tests that are also in core. Jignesh and I were talking about writing a Pole Position-style test which measures peformance on each of a couple dozen specific operations. There are limits to what we could ship with PostgreSQL ... DW operations aren't really testable without 18 hours to generate data ... but we could test a lot of things. I think we're going to need several sets of tests, and some settable scaling factors. Performance isn't just about humungous DW apps. We need to have testing on small to medium sized apps/machinery as well. cheers andrew ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Quality and Performance
Andrew Dunstan [EMAIL PROTECTED] writes: Josh Berkus wrote: ... DW operations aren't really testable without 18 hours to generate data ... but we could test a lot of things. Performance isn't just about humungous DW apps. Indeed. I think the real take-home lesson from these past few days' discussion is that *any* particular view of performance is going to miss things that don't affect that case, but do affect somebody else. What I find most worrisome about the notion of setting up a performance-farm is that it will encourage us to optimize with blinkers on --- that is, that we will consider only the specific cases measured by whatever tests are included in the farm, and will happily pessimize other cases. We can ameliorate that a bit if we can get a sufficiently wide variety of test cases, but it will always be a concern. And dogmatic positions like only cases involving terabytes of data are worth testing are definitely not going to help. regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Quality and Performance
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wed, 28 Nov 2007 00:15:48 -0500 Tom Lane [EMAIL PROTECTED] wrote: Andrew Dunstan [EMAIL PROTECTED] writes: Josh Berkus wrote: ... DW operations aren't really testable without 18 hours to generate data ... but we could test a lot of things. Performance isn't just about humungous DW apps. Indeed. I think the real take-home lesson from these past few days' discussion is that *any* particular view of performance is going to miss things that don't affect that case, but do affect somebody else. What I find most worrisome about the notion of setting up a performance-farm is that it will encourage us to optimize with blinkers on --- that is, that we will consider only the specific cases measured by whatever tests are included in the farm, and will happily pessimize other cases. We can ameliorate that a bit if we can get a sufficiently wide variety of test cases, but it will always be a concern. And dogmatic positions like only cases involving terabytes of data are worth testing are definitely not going to help. Well I certainly agree with that, especially considering that although we do have installations with that type of data, the percentage is microscopic compared to those that don't. I think it may be interested to host a series of different test from pgbench, odbcbench, dbt2, custom scripts etc... to provide trending for a particular host. That way if one host does 50tps continuously and then changes one way or the other, there is at least a flag... Sincerely, Joshua D. Drake - -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240 PostgreSQL solutions since 1997 http://www.commandprompt.com/ UNIQUE NOT NULL Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate PostgreSQL Replication: http://www.commandprompt.com/products/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFHTP3iATb/zqfZUUQRAgQ3AJ9N71Xgl+O4H4dr/IFagzjxu0HvAwCfZVkB ReQmGxacCqilRstLGhoqBU4= =03BG -END PGP SIGNATURE- ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Quality and Performance
Tom Lane wrote: Andrew Dunstan [EMAIL PROTECTED] writes: Josh Berkus wrote: ... DW operations aren't really testable without 18 hours to generate data ... but we could test a lot of things. Performance isn't just about humungous DW apps. Indeed. I think the real take-home lesson from these past few days' discussion is that *any* particular view of performance is going to miss things that don't affect that case, but do affect somebody else. yep - but the do not affect somebody else might be one that could be actually catched by something like a benchfarm. What I find most worrisome about the notion of setting up a performance-farm is that it will encourage us to optimize with blinkers on --- that is, that we will consider only the specific cases measured by whatever tests are included in the farm, and will happily pessimize other cases. We can ameliorate that a bit if we can get a sufficiently wide variety of test cases, but it will always be a concern. And dogmatic positions like only cases involving terabytes of data are worth testing are definitely not going to help. agreed - I don't think having the tests itself in core (at least initially) is such a good idea(neither am I sure tacking it on top of the buildfarm really is). There are a LOT of things we could do with such a farm/infrastructure but it will take time to exactly figure out what we can reasonably do on an automated/regular base and in a common framework. Stefan ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq