Re: [HACKERS] How can we make beta testing better?

2014-05-09 Thread Josh Berkus
On 04/26/2014 12:39 PM, Jim Nasby wrote: On 4/17/14, 6:42 PM, Josh Berkus wrote: So we have some software we've been procrastinating on OSS'ing, which does: 1) Takes full query CSV logs from a running postgres instance 2) Runs them against a target instance in parallel 3) Records response

Re: [HACKERS] How can we make beta testing better?

2014-04-26 Thread Jim Nasby
On 4/17/14, 6:42 PM, Josh Berkus wrote: So we have some software we've been procrastinating on OSS'ing, which does: 1) Takes full query CSV logs from a running postgres instance 2) Runs them against a target instance in parallel 3) Records response times for all queries Is that the stuff

Re: [HACKERS] How can we make beta testing better?

2014-04-22 Thread Jehan-Guillaume de Rorthais
On Thu, 17 Apr 2014 16:42:21 -0700 Josh Berkus j...@agliodbs.com wrote: On 04/15/2014 09:53 PM, Rod Taylor wrote: A documented beta test process/toolset which does the following would help: 1) Enables full query logging 2) Creates a replica of a production DB, record $TIME when it stops.

Re: [HACKERS] How can we make beta testing better?

2014-04-18 Thread MauMau
From: Josh Berkus j...@agliodbs.com How can we make beta testing better and more effective? How can we get more users to actually throw serious workloads at new versions and share the results? I've tried a couple of things over the last two years and they haven't worked all that well. Since

Re: [HACKERS] How can we make beta testing better?

2014-04-18 Thread Alvaro Herrera
MauMau wrote: From: Josh Berkus j...@agliodbs.com How can we make beta testing better and more effective? How can we get more users to actually throw serious workloads at new versions and share the results? I've tried a couple of things over the last two years and they haven't worked all

Re: [HACKERS] How can we make beta testing better?

2014-04-18 Thread Greg Stark
On Fri, Apr 18, 2014 at 4:15 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: There was no reply, which I took as this isn't a new feature and isn't user visible anyway, so what would be the point? To be fair the list was pretty long already. And like regression testing, coming up with a

Re: [HACKERS] How can we make beta testing better?

2014-04-18 Thread Josh Berkus
On 04/18/2014 08:15 AM, Alvaro Herrera wrote: and see whether it still works at all for you. I asked Josh specifically to mention it in a followup to this message which you can see in that thread. There was no reply, which I took as this isn't a new feature and isn't user visible anyway, so

Re: [HACKERS] How can we make beta testing better?

2014-04-17 Thread Greg Stark
On Wed, Apr 16, 2014 at 12:53 AM, Rod Taylor rod.tay...@gmail.com wrote: 4) Plays queries from the CSV logs starting from $TIME mimicking actual timing and transaction boundaries This ^^ But I recall a number of previous attempts including plugins for general load testing systems, what

Re: [HACKERS] How can we make beta testing better?

2014-04-17 Thread Josh Berkus
On 04/17/2014 05:39 AM, Greg Stark wrote: On Wed, Apr 16, 2014 at 12:53 AM, Rod Taylor rod.tay...@gmail.com wrote: 4) Plays queries from the CSV logs starting from $TIME mimicking actual timing and transaction boundaries This ^^ But I recall a number of previous attempts including plugins

Re: [HACKERS] How can we make beta testing better?

2014-04-17 Thread Merlin Moncure
On Tue, Apr 15, 2014 at 4:47 PM, Josh Berkus j...@agliodbs.com wrote: Hackers, I think 9.3 has given us evidence that our users aren't giving new versions of PostgreSQL substantial beta testing, or if they are, they aren't sharing the results with us. How can we make beta testing better and

Re: [HACKERS] How can we make beta testing better?

2014-04-17 Thread Jeff Janes
On Tue, Apr 15, 2014 at 2:47 PM, Josh Berkus j...@agliodbs.com wrote: Hackers, I think 9.3 has given us evidence that our users aren't giving new versions of PostgreSQL substantial beta testing, or if they are, they aren't sharing the results with us. A lot of the bugs that turned up are

Re: [HACKERS] How can we make beta testing better?

2014-04-17 Thread Jan Wieck
On 04/17/14 15:16, Merlin Moncure wrote: On Tue, Apr 15, 2014 at 4:47 PM, Josh Berkus j...@agliodbs.com wrote: Hackers, I think 9.3 has given us evidence that our users aren't giving new versions of PostgreSQL substantial beta testing, or if they are, they aren't sharing the results with us.

Re: [HACKERS] How can we make beta testing better?

2014-04-17 Thread Josh Berkus
On 04/15/2014 09:53 PM, Rod Taylor wrote: A documented beta test process/toolset which does the following would help: 1) Enables full query logging 2) Creates a replica of a production DB, record $TIME when it stops. 3) Allow user to make changes (upgrade to 9.4, change hardware, change

Re: [HACKERS] How can we make beta testing better?

2014-04-17 Thread Greg Stark
On Thu, Apr 17, 2014 at 5:26 PM, Jeff Janes jeff.ja...@gmail.com wrote: A lot of the bugs that turned up are not the kind I would expect to have been found in most beta testing done by non-hacking users. Weren't they mostly around rare race conditions, crash recovery, and freezing? Actually I

[HACKERS] How can we make beta testing better?

2014-04-15 Thread Josh Berkus
Hackers, I think 9.3 has given us evidence that our users aren't giving new versions of PostgreSQL substantial beta testing, or if they are, they aren't sharing the results with us. How can we make beta testing better and more effective? How can we get more users to actually throw serious

Re: [HACKERS] How can we make beta testing better?

2014-04-15 Thread Rod Taylor
On Tue, Apr 15, 2014 at 5:47 PM, Josh Berkus j...@agliodbs.com wrote: Hackers, I think 9.3 has given us evidence that our users aren't giving new versions of PostgreSQL substantial beta testing, or if they are, they aren't sharing the results with us. How can we make beta testing better